2024-03-16T19:36:34.242 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T19:36:34.242 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --name leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi --description fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} --verbose --archive /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118 -- /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/orig.config.yaml 2024-03-16T19:36:34.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T19:36:34.376 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118 branch: wip-lusov-qdb-squid description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} email: null first_in_suite: false job_id: '7602118' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-qdb-squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow request - SLOW_OPS - MON_DOWN sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b valgrind: exit_on_first_error: false mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb-squid sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 6918 sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb-squid suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZw9i/4Vc9mop3CJjhqkAmxhPoQx9KrKTXVtPi6nmyvo7tsBHjJcg/SGsfC7PUNI7tjtIVIhxU0ZRID3L2wjFk= smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtaE+d2gd3u/ZI1NLSrq7vqbsLHzfwJBx7MfraW5/9rpHJZc1isETXaFsfab+PC6JveTnZ0pPQTlW8a5YAxKgY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_11:44:52 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T19:36:34.376 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa; will attempt to use it 2024-03-16T19:36:34.377 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks 2024-03-16T19:36:34.377 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T19:36:34.379 INFO:teuthology.task.internal:Checking packages... 2024-03-16T19:36:34.401 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b' 2024-03-16T19:36:34.401 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T19:36:34.401 INFO:teuthology.packaging:ref: None 2024-03-16T19:36:34.401 INFO:teuthology.packaging:tag: None 2024-03-16T19:36:34.401 INFO:teuthology.packaging:branch: wip-lusov-qdb-squid 2024-03-16T19:36:34.401 INFO:teuthology.packaging:sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:36:34.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-qdb-squid 2024-03-16T19:36:34.702 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:36:34.704 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T19:36:34.712 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T19:36:34.712 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T19:36:34.759 INFO:teuthology.task.internal:Saving configuration 2024-03-16T19:36:34.775 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T19:36:34.787 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T19:36:34.812 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:31:07.741885', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB54DgpqS2uLnhqREoZHGX+41zEdp5wtf0ixDWBRCBeV'} 2024-03-16T19:36:34.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:31:07.739957', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILQbr3JjQfjNolEuwrKFBmYnUL5UYo1wCNwUDz0kQpXY'} 2024-03-16T19:36:34.837 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T19:36:34.843 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-16T19:36:34.843 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-16T19:36:34.843 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T19:36:34.936 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8801b0d3a0>, signals=[15]) 2024-03-16T19:36:34.936 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T19:36:34.943 INFO:teuthology.task.internal:Opening connections... 2024-03-16T19:36:34.943 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:36:34.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:36:35.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:36:35.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:36:35.117 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T19:36:35.126 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-03-16T19:36:35.132 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-03-16T19:36:35.132 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu" 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04" 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian 2024-03-16T19:36:35.179 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-16T19:36:35.180 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-16T19:36:35.180 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-16T19:36:35.180 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-16T19:36:35.180 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy 2024-03-16T19:36:35.180 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-03-16T19:36:35.201 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-03-16T19:36:35.207 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-03-16T19:36:35.207 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-03-16T19:36:35.254 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="22.04" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=jammy 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-16T19:36:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=jammy 2024-03-16T19:36:35.255 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-16T19:36:35.276 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T19:36:35.285 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T19:36:35.310 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T19:36:35.310 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-03-16T19:36:35.312 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-03-16T19:36:35.317 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T19:36:35.343 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T19:36:35.343 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-03-16T19:36:35.360 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-03-16T19:36:35.377 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T19:36:35.431 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T19:36:35.445 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-16T19:36:35.445 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-16T19:36:35.445 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T19:36:35.445 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T19:36:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:36:35.450 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-16T19:36:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:36:35.451 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-16T19:36:35.500 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-56-generic 2024-03-16T19:36:35.500 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-56-generic 2024-03-16T19:36:35.500 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.15.0-56-generic 2024-03-16T19:36:35.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-03-16T19:36:35.500 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.15.0-56-generic 2024-03-16T19:36:35.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-03-16T19:36:35.614 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-03-16T19:36:35.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-03-16T19:36:35.841 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:36:35.850 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:36:35.850 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-16T19:36:35.863 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-16T19:36:35.876 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-16T19:36:35.909 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-16T19:36:36.069 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-16T19:36:36.078 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-16T19:36:36.138 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-16T19:36:36.189 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [587 kB] 2024-03-16T19:36:36.218 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [587 kB] 2024-03-16T19:36:36.378 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,462 kB] 2024-03-16T19:36:36.384 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-16T19:36:36.452 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-16T19:36:36.454 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,462 kB] 2024-03-16T19:36:36.539 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-16T19:36:36.544 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-16T19:36:36.545 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-16T19:36:36.546 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-16T19:36:36.558 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [285 kB] 2024-03-16T19:36:36.580 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-16T19:36:36.587 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-16T19:36:36.598 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-16T19:36:36.618 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-16T19:36:36.618 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [598 kB] 2024-03-16T19:36:36.662 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-16T19:36:36.706 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-16T19:36:36.708 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-16T19:36:36.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [285 kB] 2024-03-16T19:36:36.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [850 kB] 2024-03-16T19:36:36.720 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-16T19:36:36.720 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-16T19:36:36.725 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-16T19:36:36.726 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [695 kB] 2024-03-16T19:36:36.728 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-16T19:36:36.731 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-16T19:36:36.732 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-16T19:36:36.733 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-16T19:36:36.733 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-16T19:36:36.733 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-16T19:36:36.760 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,057 kB] 2024-03-16T19:36:36.810 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-16T19:36:36.841 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-16T19:36:36.843 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-16T19:36:36.859 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-16T19:36:36.859 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,057 kB] 2024-03-16T19:36:36.906 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [695 kB] 2024-03-16T19:36:36.931 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-16T19:36:36.943 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-16T19:36:36.947 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-16T19:36:36.948 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-16T19:36:36.949 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-16T19:36:36.950 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-16T19:36:36.950 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-16T19:36:36.952 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-16T19:36:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-16T19:36:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-16T19:36:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-16T19:36:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-16T19:36:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-16T19:36:36.961 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-16T19:36:36.961 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-16T19:36:36.961 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-16T19:36:36.963 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-16T19:36:36.963 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-16T19:36:36.964 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-16T19:36:36.964 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-16T19:36:36.966 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-16T19:36:36.974 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-16T19:36:36.974 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-16T19:36:36.994 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-16T19:36:36.997 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-16T19:36:36.999 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-16T19:36:36.999 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [598 kB] 2024-03-16T19:36:37.009 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-16T19:36:37.009 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [850 kB] 2024-03-16T19:36:37.010 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-16T19:36:37.010 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.6 kB] 2024-03-16T19:36:37.011 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-16T19:36:37.037 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.6 kB] 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-16T19:36:37.190 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-16T19:36:37.585 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-16T19:36:37.592 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-16T19:36:37.592 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-16T19:36:37.607 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-16T19:36:37.609 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-16T19:36:42.253 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 12.2 MB in 2s (5,727 kB/s) 2024-03-16T19:36:42.573 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 12.2 MB in 2s (5,282 kB/s) 2024-03-16T19:36:43.608 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:36:43.625 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-16T19:36:43.679 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:36:43.856 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-03-16T19:36:43.857 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-03-16T19:36:43.969 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:36:43.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-16T19:36:44.014 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout: python2.7-minimal 2024-03-16T19:36:44.015 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:36:44.016 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-03-16T19:36:44.016 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-16T19:36:44.016 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-16T19:36:44.017 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-16T19:36:44.018 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-03-16T19:36:44.018 INFO:teuthology.orchestra.run.smithi059.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-16T19:36:44.018 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-03-16T19:36:44.018 INFO:teuthology.orchestra.run.smithi059.stdout: thermald 2024-03-16T19:36:44.040 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:36:44.063 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-03-16T19:36:44.063 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-16T19:36:44.064 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-16T19:36:44.064 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-16T19:36:44.065 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages will be upgraded: 2024-03-16T19:36:44.065 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-16T19:36:44.151 INFO:teuthology.orchestra.run.smithi059.stdout:3 upgraded, 5 newly installed, 0 to remove and 297 not upgraded. 2024-03-16T19:36:44.151 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 113 MB of archives. 2024-03-16T19:36:44.151 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-16T19:36:44.151 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-16T19:36:44.218 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-03-16T19:36:44.219 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-03-16T19:36:44.380 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:36:44.380 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:36:44.380 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:36:44.380 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:36:44.381 INFO:teuthology.orchestra.run.smithi032.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-16T19:36:44.381 INFO:teuthology.orchestra.run.smithi032.stdout: python2.7-minimal 2024-03-16T19:36:44.381 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:36:44.381 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-03-16T19:36:44.381 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-16T19:36:44.382 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-16T19:36:44.382 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-16T19:36:44.383 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-03-16T19:36:44.383 INFO:teuthology.orchestra.run.smithi032.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-16T19:36:44.383 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-03-16T19:36:44.383 INFO:teuthology.orchestra.run.smithi032.stdout: thermald 2024-03-16T19:36:44.429 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-03-16T19:36:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-16T19:36:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-16T19:36:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-16T19:36:44.430 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages will be upgraded: 2024-03-16T19:36:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-16T19:36:44.558 INFO:teuthology.orchestra.run.smithi032.stdout:3 upgraded, 5 newly installed, 0 to remove and 297 not upgraded. 2024-03-16T19:36:44.558 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 113 MB of archives. 2024-03-16T19:36:44.558 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-16T19:36:44.559 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-16T19:36:44.928 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-16T19:36:45.275 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-16T19:36:45.884 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-16T19:36:46.294 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-16T19:36:47.199 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-16T19:36:47.199 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-16T19:36:47.199 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-16T19:36:47.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-16T19:36:47.843 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-16T19:36:48.278 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 113 MB in 4s (30.0 MB/s) 2024-03-16T19:36:48.536 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-16T19:36:48.599 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-16T19:36:48.600 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-16T19:36:48.600 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-16T19:36:48.767 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-16T19:36:48.803 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-16T19:36:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 113 MB in 4s (25.9 MB/s) 2024-03-16T19:36:49.373 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-16T19:36:50.554 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-16T19:36:50.558 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:36:51.366 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-16T19:36:51.371 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:51.454 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:36:53.167 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-16T19:36:53.188 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:53.294 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:36:53.753 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-16T19:36:53.773 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:53.781 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-16T19:36:53.804 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:53.811 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:36:53.896 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:36:54.335 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-16T19:36:54.356 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:36:54.393 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:00.079 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:00.203 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:00.539 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:00.665 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:00.968 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-16T19:37:00.990 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-16T19:37:01.026 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-16T19:37:01.376 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:01.501 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:01.792 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:01.921 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:02.199 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-16T19:37:02.222 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-16T19:37:02.266 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-16T19:37:06.392 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-16T19:37:06.429 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:37:06.466 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:07.954 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-16T19:37:07.991 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-16T19:37:08.020 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:09.662 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:09.784 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:10.117 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-16T19:37:10.238 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:10.364 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-16T19:37:10.476 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:11.825 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:11.825 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:11.825 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:11.826 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:12.008 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-16T19:37:12.086 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:12.157 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-16T19:37:12.499 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-16T19:37:12.629 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:12.754 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-16T19:37:12.892 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:13.637 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:14.274 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:14.274 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:14.274 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:14.274 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:14.426 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:15.063 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-16T19:37:15.189 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-16T19:37:15.299 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-16T19:37:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:15.903 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:17.888 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-16T19:37:18.022 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-16T19:37:18.157 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-16T19:37:18.220 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-16T19:37:18.220 INFO:teuthology.orchestra.run.smithi032.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:30.534 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-16T19:37:30.534 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub' 2024-03-16T19:37:30.544 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-16T19:37:30.572 INFO:teuthology.orchestra.run.smithi059.stdout:Generating grub configuration file ... 2024-03-16T19:37:30.854 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:30.877 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:31.177 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:31.182 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:31.254 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-16T19:37:31.259 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-16T19:37:31.413 INFO:teuthology.orchestra.run.smithi059.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-16T19:37:31.413 INFO:teuthology.orchestra.run.smithi059.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-16T19:37:31.413 INFO:teuthology.orchestra.run.smithi059.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-16T19:37:31.461 INFO:teuthology.orchestra.run.smithi059.stdout:done 2024-03-16T19:37:31.959 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-03-16T19:37:31.980 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-03-16T19:37:31.980 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.100.97 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-03-16T19:37:31.981 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-03-16T19:37:31.982 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-16T19:37:31.982 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T19:37:31.982 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T19:37:31.982 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-16T19:37:31.983 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T19:37:31.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-03-16T19:37:32.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-03-16T19:37:32.312 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-16T19:37:32.390 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:37:32.472 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-16T19:37:32.554 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-16T19:37:32.559 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-16T19:37:32.559 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub' 2024-03-16T19:37:32.567 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-16T19:37:32.590 INFO:teuthology.orchestra.run.smithi032.stdout:Generating grub configuration file ... 2024-03-16T19:37:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:32.942 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:33.224 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:33.229 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:33.289 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-16T19:37:33.294 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-16T19:37:33.451 INFO:teuthology.orchestra.run.smithi032.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-16T19:37:33.451 INFO:teuthology.orchestra.run.smithi032.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-16T19:37:33.452 INFO:teuthology.orchestra.run.smithi032.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-16T19:37:33.502 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-03-16T19:37:33.728 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:37:33.745 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-16T19:37:33.806 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:37:33.970 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-03-16T19:37:34.001 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-03-16T19:37:34.001 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-03-16T19:37:34.001 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.100.97 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-03-16T19:37:34.002 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-03-16T19:37:34.002 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-16T19:37:34.003 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T19:37:34.003 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T19:37:34.003 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-16T19:37:34.003 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T19:37:34.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-03-16T19:37:34.003 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-03-16T19:37:34.004 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-03-16T19:37:34.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:37:34.187 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:37:34.188 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:37:34.188 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:37:34.230 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 297 not upgraded. 2024-03-16T19:37:34.232 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.100.97 2024-03-16T19:37:34.248 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-16T19:37:34.249 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-16T19:37:34.249 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-03-16T19:37:34.249 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-03-16T19:37:34.249 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-03-16T19:37:34.249 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-03-16T19:37:34.249 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-16T19:37:34.294 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.m3l0DTDU9Z 2024-03-16T19:37:34.294 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub/grub.cfg /tmp/tmp.m3l0DTDU9Z 2024-03-16T19:37:34.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.m3l0DTDU9Z 2024-03-16T19:37:34.426 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:37:34.426 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-16T19:37:34.472 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.m3l0DTDU9Z is 10KB 2024-03-16T19:37:34.485 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.m3l0DTDU9Z 2024-03-16T19:37:34.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-16T19:37:34.498 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-16T19:37:34.546 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:37:34.546 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-16T19:37:34.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-16T19:37:34.579 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-16T19:37:34.614 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-16T19:37:34.615 DEBUG:teuthology.orchestra.run.smithi059:> sudo update-grub 2024-03-16T19:37:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub' 2024-03-16T19:37:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-16T19:37:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-03-16T19:37:35.825 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:37:35.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-16T19:37:35.906 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:37:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:35.982 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-03-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-03-16T19:37:36.218 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:36.222 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-16T19:37:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-16T19:37:36.281 INFO:teuthology.orchestra.run.smithi032.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-16T19:37:36.281 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:37:36.281 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:37:36.281 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:37:36.281 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:37:36.282 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:37:36.282 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:37:36.282 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:37:36.324 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 0 newly installed, 0 to remove and 297 not upgraded. 2024-03-16T19:37:36.326 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.100.97 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-03-16T19:37:36.343 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-03-16T19:37:36.344 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-03-16T19:37:36.344 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-16T19:37:36.390 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.PHPxLazbET 2024-03-16T19:37:36.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub/grub.cfg /tmp/tmp.PHPxLazbET 2024-03-16T19:37:36.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.PHPxLazbET 2024-03-16T19:37:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-16T19:37:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-16T19:37:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-16T19:37:36.526 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-03-16T19:37:36.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-03-16T19:37:36.566 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.PHPxLazbET is 10KB 2024-03-16T19:37:36.579 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.PHPxLazbET 2024-03-16T19:37:36.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-16T19:37:36.638 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:37:36.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-16T19:37:36.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-16T19:37:36.740 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-16T19:37:36.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo update-grub 2024-03-16T19:37:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub' 2024-03-16T19:37:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-16T19:37:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-03-16T19:37:38.101 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-16T19:37:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-16T19:37:38.337 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-16T19:37:38.342 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-16T19:37:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-16T19:37:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-16T19:37:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-16T19:37:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-16T19:37:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-16T19:37:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-03-16T19:37:38.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-03-16T19:38:06.570 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-16T19:38:06.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-16T19:38:06.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:38:08.537 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-16T19:38:08.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-16T19:38:08.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:39:06.631 DEBUG:teuthology.orchestra.remote:timed out 2024-03-16T19:39:08.542 DEBUG:teuthology.orchestra.remote:timed out 2024-03-16T19:39:15.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-16T19:39:15.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:39:17.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-16T19:39:17.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:39:18.046 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-16T19:39:19.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-16T19:39:19.399 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-100-generic"... 2024-03-16T19:39:19.399 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-16T19:39:19.452 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-100-generic 2024-03-16T19:39:19.453 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-16T19:39:19.453 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T19:39:19.453 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-16T19:39:20.454 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-16T19:39:20.455 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T19:39:20.687 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-03-16T19:39:20.721 DEBUG:teuthology.parallel:result is None 2024-03-16T19:39:31.194 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-16T19:39:31.902 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-16T19:39:31.902 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-100-generic"... 2024-03-16T19:39:31.903 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-16T19:39:31.955 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-100-generic 2024-03-16T19:39:31.955 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-16T19:39:31.955 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T19:39:31.955 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-16T19:39:32.956 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-16T19:39:32.956 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T19:39:33.019 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-03-16T19:39:33.055 DEBUG:teuthology.parallel:result is None 2024-03-16T19:39:33.055 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T19:39:33.062 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T19:39:33.062 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T19:39:33.065 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T19:39:33.070 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T19:39:33.097 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T19:39:33.136 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T19:39:33.136 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T19:39:33.138 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T19:39:33.158 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T19:39:33.164 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T19:39:33.164 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T19:39:33.193 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T19:39:33.207 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:39:33.217 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:39:33.222 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:39:33.230 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:39:33.232 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T19:39:33.239 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T19:39:33.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T19:39:33.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T19:39:33.285 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T19:39:33.294 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T19:39:33.295 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T19:39:33.324 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T19:39:33.333 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:39:33.376 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:39:33.424 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:39:33.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T19:39:33.485 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:39:33.491 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:39:33.538 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:39:33.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T19:39:33.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-03-16T19:39:33.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-03-16T19:39:33.677 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T19:39:33.684 INFO:teuthology.task.internal:Starting timer... 2024-03-16T19:39:33.685 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T19:39:33.712 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T19:39:33.744 DEBUG:teuthology.task.selinux:Excluding smithi032: OS 'ubuntu' does not support SELinux 2024-03-16T19:39:33.745 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2024-03-16T19:39:33.745 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T19:39:33.745 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-16T19:39:33.745 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T19:39:33.745 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T19:39:33.778 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T19:39:33.795 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T19:39:33.797 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T19:46:38.885 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2024-03-16T19:46:38.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-16T19:46:38.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:46:38.969 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-16T19:46:39.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-16T19:46:39.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-16T19:46:39.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:46:39.130 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-16T19:46:39.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-16T19:46:39.207 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T19:46:39.218 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T19:46:39.219 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T19:46:39.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T19:46:39.222 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T19:46:39.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Command line: ntpd -gq 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: ---------------------------------------------------- 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: ntp-4 is maintained by Network Time Foundation, 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: corporation. Support and training for ntp-4 are 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: available at https://www.nwtime.org/support 2024-03-16T19:46:39.249 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: ---------------------------------------------------- 2024-03-16T19:46:39.250 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: proto: precision = 0.071 usec (-24) 2024-03-16T19:46:39.250 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: basedate set to 2022-02-04 2024-03-16T19:46:39.251 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: gps base set to 2022-02-06 (week 2196) 2024-03-16T19:46:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-16T19:46:39.251 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-16T19:46:39.251 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: restrict ::: KOD does nothing without LIMITED. 2024-03-16T19:46:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-16T19:46:39.252 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen and drop on 0 v6wildcard [::]:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen normally on 2 lo 127.0.0.1:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen normally on 3 ens1f0 172.21.15.32:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen normally on 4 lo [::1]:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2602%4]:123 2024-03-16T19:46:39.253 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:39 ntpd[17122]: Listening on routing socket on fd #22 for interface updates 2024-03-16T19:46:39.281 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Command line: ntpd -gq 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: ---------------------------------------------------- 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: ntp-4 is maintained by Network Time Foundation, 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: corporation. Support and training for ntp-4 are 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: available at https://www.nwtime.org/support 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: ---------------------------------------------------- 2024-03-16T19:46:39.282 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: proto: precision = 0.046 usec (-24) 2024-03-16T19:46:39.283 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: basedate set to 2022-02-04 2024-03-16T19:46:39.283 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: gps base set to 2022-02-06 (week 2196) 2024-03-16T19:46:39.283 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-16T19:46:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-16T19:46:39.284 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: restrict ::: KOD does nothing without LIMITED. 2024-03-16T19:46:39.284 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen and drop on 0 v6wildcard [::]:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen normally on 2 lo 127.0.0.1:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen normally on 3 ens1f0 172.21.15.59:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen normally on 4 lo [::1]:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%4]:123 2024-03-16T19:46:39.285 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:39 ntpd[17045]: Listening on routing socket on fd #22 for interface updates 2024-03-16T19:46:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:40 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:40 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:40.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:40 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:41 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:42.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:42 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:42 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:43 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:44 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:44 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:45 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:46.249 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:46 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stdout:16 Mar 19:46:46 ntpd[17045]: ntpd: time slew +0.000408 s 2024-03-16T19:46:46.284 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time slew +0.000408s 2024-03-16T19:46:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-16T19:46:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:16 Mar 19:46:46 ntpd[17045]: can't open /var/log/ntpstats/loopstats.20240316: Permission denied 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:46.344 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: can't open /var/log/ntpstats/rawstats.20240316: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-16T19:46:47.250 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: can't open /var/log/ntpstats/peerstats.20240316: Permission denied 2024-03-16T19:46:47.251 INFO:teuthology.orchestra.run.smithi032.stdout:16 Mar 19:46:47 ntpd[17122]: ntpd: time slew +0.000188 s 2024-03-16T19:46:47.251 INFO:teuthology.orchestra.run.smithi032.stdout:ntpd: time slew +0.000188s 2024-03-16T19:46:47.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-16T19:46:47.251 INFO:teuthology.orchestra.run.smithi032.stderr:16 Mar 19:46:47 ntpd[17122]: can't open /var/log/ntpstats/loopstats.20240316: Permission denied 2024-03-16T19:46:47.300 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-03-16T19:46:47.300 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-03-16T19:46:47.300 INFO:teuthology.orchestra.run.smithi032.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:47.300 INFO:teuthology.orchestra.run.smithi032.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:47.301 INFO:teuthology.orchestra.run.smithi032.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:47.301 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-16T19:46:47.302 INFO:teuthology.run_tasks:Running task install... 2024-03-16T19:46:47.311 DEBUG:teuthology.task.install:project ceph 2024-03-16T19:46:47.311 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}} 2024-03-16T19:46:47.311 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'} 2024-03-16T19:46:47.311 INFO:teuthology.task.install:Using flavor: default 2024-03-16T19:46:47.317 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-16T19:46:47.317 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T19:46:47.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-key list | grep Ceph 2024-03-16T19:46:47.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-key list | grep Ceph 2024-03-16T19:46:47.386 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-16T19:46:47.404 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-16T19:46:47.417 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-16T19:46:47.417 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph.com (release key) 2024-03-16T19:46:47.418 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, ceph-common-dbg, ceph-fuse-dbg, ceph-mds-dbg, ceph-mgr-dbg, ceph-mon-dbg, ceph-osd-dbg, ceph-test-dbg, libcephfs2-dbg, librados2-dbg, libradosstriper1-dbg, librbd1-dbg, librgw2-dbg, radosgw-dbg, rbd-fuse-dbg, rbd-mirror-dbg, rbd-nbd-dbg, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-16T19:46:47.419 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-16T19:46:47.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:46:47.436 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-16T19:46:47.436 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph.com (release key) 2024-03-16T19:46:47.437 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, ceph-common-dbg, ceph-fuse-dbg, ceph-mds-dbg, ceph-mgr-dbg, ceph-mon-dbg, ceph-osd-dbg, ceph-test-dbg, libcephfs2-dbg, librados2-dbg, libradosstriper1-dbg, librbd1-dbg, librgw2-dbg, radosgw-dbg, rbd-fuse-dbg, rbd-mirror-dbg, rbd-nbd-dbg, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-16T19:46:47.437 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-16T19:46:47.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:46:47.560 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default/ 2024-03-16T19:46:47.560 INFO:teuthology.task.install.deb:Package version is 19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:46:47.574 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default/ 2024-03-16T19:46:47.574 INFO:teuthology.task.install.deb:Package version is 19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:46:47.682 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:46:47.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-16T19:46:47.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-03-16T19:46:47.735 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:46:47.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-16T19:46:47.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-03-16T19:46:47.875 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-16T19:46:47.923 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-16T19:46:47.944 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-16T19:46:47.966 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-16T19:46:47.987 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-16T19:46:48.015 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,395 B] 2024-03-16T19:46:48.036 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-16T19:46:48.051 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:46:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-16T19:46:48.110 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-16T19:46:48.123 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-16T19:46:48.146 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,395 B] 2024-03-16T19:46:48.189 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-16T19:46:48.205 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-16T19:46:48.490 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-16T19:46:48.544 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-16T19:46:48.571 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-16T19:46:48.653 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-16T19:46:48.720 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 28.9 kB in 1s (37.3 kB/s) 2024-03-16T19:46:48.818 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 28.9 kB in 1s (30.8 kB/s) 2024-03-16T19:46:49.667 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:46:49.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1681-gaba3c1d2-1jammy cephadm=19.0.0-1681-gaba3c1d2-1jammy ceph-mds=19.0.0-1681-gaba3c1d2-1jammy ceph-mgr=19.0.0-1681-gaba3c1d2-1jammy ceph-common=19.0.0-1681-gaba3c1d2-1jammy ceph-fuse=19.0.0-1681-gaba3c1d2-1jammy ceph-test=19.0.0-1681-gaba3c1d2-1jammy ceph-volume=19.0.0-1681-gaba3c1d2-1jammy radosgw=19.0.0-1681-gaba3c1d2-1jammy python3-rados=19.0.0-1681-gaba3c1d2-1jammy python3-rgw=19.0.0-1681-gaba3c1d2-1jammy python3-cephfs=19.0.0-1681-gaba3c1d2-1jammy python3-rbd=19.0.0-1681-gaba3c1d2-1jammy libcephfs2=19.0.0-1681-gaba3c1d2-1jammy libcephfs-dev=19.0.0-1681-gaba3c1d2-1jammy librados2=19.0.0-1681-gaba3c1d2-1jammy librbd1=19.0.0-1681-gaba3c1d2-1jammy rbd-fuse=19.0.0-1681-gaba3c1d2-1jammy ceph-common-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-fuse-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mds-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mgr-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mon-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-osd-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-test-dbg=19.0.0-1681-gaba3c1d2-1jammy libcephfs2-dbg=19.0.0-1681-gaba3c1d2-1jammy librados2-dbg=19.0.0-1681-gaba3c1d2-1jammy libradosstriper1-dbg=19.0.0-1681-gaba3c1d2-1jammy librbd1-dbg=19.0.0-1681-gaba3c1d2-1jammy librgw2-dbg=19.0.0-1681-gaba3c1d2-1jammy radosgw-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-fuse-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-mirror-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-nbd-dbg=19.0.0-1681-gaba3c1d2-1jammy python3-cephfs=19.0.0-1681-gaba3c1d2-1jammy cephfs-shell=19.0.0-1681-gaba3c1d2-1jammy cephfs-top=19.0.0-1681-gaba3c1d2-1jammy cephfs-mirror=19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:46:49.735 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:46:49.780 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:46:49.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1681-gaba3c1d2-1jammy cephadm=19.0.0-1681-gaba3c1d2-1jammy ceph-mds=19.0.0-1681-gaba3c1d2-1jammy ceph-mgr=19.0.0-1681-gaba3c1d2-1jammy ceph-common=19.0.0-1681-gaba3c1d2-1jammy ceph-fuse=19.0.0-1681-gaba3c1d2-1jammy ceph-test=19.0.0-1681-gaba3c1d2-1jammy ceph-volume=19.0.0-1681-gaba3c1d2-1jammy radosgw=19.0.0-1681-gaba3c1d2-1jammy python3-rados=19.0.0-1681-gaba3c1d2-1jammy python3-rgw=19.0.0-1681-gaba3c1d2-1jammy python3-cephfs=19.0.0-1681-gaba3c1d2-1jammy python3-rbd=19.0.0-1681-gaba3c1d2-1jammy libcephfs2=19.0.0-1681-gaba3c1d2-1jammy libcephfs-dev=19.0.0-1681-gaba3c1d2-1jammy librados2=19.0.0-1681-gaba3c1d2-1jammy librbd1=19.0.0-1681-gaba3c1d2-1jammy rbd-fuse=19.0.0-1681-gaba3c1d2-1jammy ceph-common-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-fuse-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mds-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mgr-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-mon-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-osd-dbg=19.0.0-1681-gaba3c1d2-1jammy ceph-test-dbg=19.0.0-1681-gaba3c1d2-1jammy libcephfs2-dbg=19.0.0-1681-gaba3c1d2-1jammy librados2-dbg=19.0.0-1681-gaba3c1d2-1jammy libradosstriper1-dbg=19.0.0-1681-gaba3c1d2-1jammy librbd1-dbg=19.0.0-1681-gaba3c1d2-1jammy librgw2-dbg=19.0.0-1681-gaba3c1d2-1jammy radosgw-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-fuse-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-mirror-dbg=19.0.0-1681-gaba3c1d2-1jammy rbd-nbd-dbg=19.0.0-1681-gaba3c1d2-1jammy python3-cephfs=19.0.0-1681-gaba3c1d2-1jammy cephfs-shell=19.0.0-1681-gaba3c1d2-1jammy cephfs-top=19.0.0-1681-gaba3c1d2-1jammy cephfs-mirror=19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:46:49.858 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:46:49.911 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-03-16T19:46:49.912 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-03-16T19:46:50.065 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-03-16T19:46:50.065 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-03-16T19:46:50.083 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:46:50.083 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:46:50.083 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:46:50.083 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:46:50.083 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:46:50.084 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:46:50.084 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:46:50.084 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-03-16T19:46:50.084 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-16T19:46:50.084 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-16T19:46:50.086 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-16T19:46:50.087 INFO:teuthology.orchestra.run.smithi059.stdout: qttranslations5-l10n rbd-mirror rbd-nbd socat unzip xmlstarlet zip 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-03-16T19:46:50.088 INFO:teuthology.orchestra.run.smithi059.stdout: btrfs-tools 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: ceph ceph-base ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-mds 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-dbg ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-dbg 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon ceph-mon-dbg ceph-osd ceph-osd-dbg ceph-test ceph-test-dbg 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume cephadm cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 libcephfs2-dbg libdouble-conversion3 libjq1 liblttng-ust1 2024-03-16T19:46:50.223 INFO:teuthology.orchestra.run.smithi059.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: libqt5network5 librados2 librados2-dbg libradosstriper1 libradosstriper1-dbg 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 librbd1-dbg librdkafka1 libreadline-dev librgw2 librgw2-dbg 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-16T19:46:50.224 INFO:teuthology.orchestra.run.smithi059.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: radosgw-dbg rbd-fuse rbd-fuse-dbg rbd-mirror rbd-mirror-dbg rbd-nbd 2024-03-16T19:46:50.225 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-dbg socat unzip xmlstarlet zip 2024-03-16T19:46:50.252 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:46:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:46:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:46:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:46:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:46:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:46:50.253 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:46:50.254 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-03-16T19:46:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-16T19:46:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-16T19:46:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-16T19:46:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-16T19:46:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-16T19:46:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-16T19:46:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: qttranslations5-l10n rbd-mirror rbd-nbd socat unzip xmlstarlet zip 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-16T19:46:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-16T19:46:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-16T19:46:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-16T19:46:50.258 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-03-16T19:46:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: btrfs-tools 2024-03-16T19:46:50.287 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 142 newly installed, 0 to remove and 284 not upgraded. 2024-03-16T19:46:50.288 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 3,767 MB of archives. 2024-03-16T19:46:50.288 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 4,582 MB of additional disk space will be used. 2024-03-16T19:46:50.288 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-16T19:46:50.308 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-16T19:46:50.319 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-16T19:46:50.367 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-16T19:46:50.391 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout: ceph ceph-base ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-mds 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-dbg ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-dbg 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon ceph-mon-dbg ceph-osd ceph-osd-dbg ceph-test ceph-test-dbg 2024-03-16T19:46:50.394 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume cephadm cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev 2024-03-16T19:46:50.395 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 libcephfs2-dbg libdouble-conversion3 libjq1 liblttng-ust1 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: libqt5network5 librados2 librados2-dbg libradosstriper1 libradosstriper1-dbg 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 librbd1-dbg librdkafka1 libreadline-dev librgw2 librgw2-dbg 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-16T19:46:50.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: radosgw-dbg rbd-fuse rbd-fuse-dbg rbd-mirror rbd-mirror-dbg rbd-nbd 2024-03-16T19:46:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-dbg socat unzip xmlstarlet zip 2024-03-16T19:46:50.407 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-16T19:46:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-16T19:46:50.464 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-16T19:46:50.466 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 142 newly installed, 0 to remove and 284 not upgraded. 2024-03-16T19:46:50.466 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 3,767 MB of archives. 2024-03-16T19:46:50.466 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 4,582 MB of additional disk space will be used. 2024-03-16T19:46:50.466 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-16T19:46:50.476 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-16T19:46:50.480 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-16T19:46:50.480 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-16T19:46:50.487 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-16T19:46:50.489 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-16T19:46:50.491 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-16T19:46:50.494 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-16T19:46:50.495 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-16T19:46:50.495 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-16T19:46:50.498 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-16T19:46:50.503 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-16T19:46:50.506 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-16T19:46:50.507 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-16T19:46:50.508 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-16T19:46:50.523 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-16T19:46:50.524 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-16T19:46:50.525 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-16T19:46:50.525 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-16T19:46:50.525 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-16T19:46:50.526 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-16T19:46:50.543 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-16T19:46:50.543 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-16T19:46:50.546 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-16T19:46:50.546 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-16T19:46:50.548 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-16T19:46:50.562 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-16T19:46:50.563 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-16T19:46:50.564 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-16T19:46:50.564 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-16T19:46:50.564 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-16T19:46:50.566 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-16T19:46:50.567 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1681-gaba3c1d2-1jammy [3,546 kB] 2024-03-16T19:46:50.574 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-16T19:46:50.577 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-16T19:46:50.583 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-16T19:46:50.584 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-16T19:46:50.589 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-16T19:46:50.589 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-16T19:46:50.590 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-16T19:46:50.603 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-16T19:46:50.604 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-16T19:46:50.605 INFO:teuthology.orchestra.run.smithi059.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-16T19:46:50.607 INFO:teuthology.orchestra.run.smithi059.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-16T19:46:50.607 INFO:teuthology.orchestra.run.smithi059.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-16T19:46:50.609 INFO:teuthology.orchestra.run.smithi059.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-16T19:46:50.622 INFO:teuthology.orchestra.run.smithi059.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-16T19:46:50.625 INFO:teuthology.orchestra.run.smithi059.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-16T19:46:50.625 INFO:teuthology.orchestra.run.smithi059.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-16T19:46:50.626 INFO:teuthology.orchestra.run.smithi059.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-16T19:46:50.642 INFO:teuthology.orchestra.run.smithi059.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-16T19:46:50.648 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-16T19:46:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-16T19:46:50.666 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-16T19:46:50.673 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-16T19:46:50.674 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-16T19:46:50.685 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-16T19:46:50.688 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-16T19:46:50.690 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-16T19:46:50.692 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-16T19:46:50.692 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-16T19:46:50.711 INFO:teuthology.orchestra.run.smithi059.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-16T19:46:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-16T19:46:50.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-16T19:46:50.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-16T19:46:50.716 INFO:teuthology.orchestra.run.smithi059.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-16T19:46:50.718 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-16T19:46:50.721 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-16T19:46:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-16T19:46:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-16T19:46:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-16T19:46:50.726 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-16T19:46:50.737 INFO:teuthology.orchestra.run.smithi059.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-16T19:46:50.739 INFO:teuthology.orchestra.run.smithi059.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-16T19:46:50.739 INFO:teuthology.orchestra.run.smithi059.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-16T19:46:50.741 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-16T19:46:50.741 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-16T19:46:50.742 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-16T19:46:50.743 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-16T19:46:50.746 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1681-gaba3c1d2-1jammy [3,546 kB] 2024-03-16T19:46:50.755 INFO:teuthology.orchestra.run.smithi059.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-16T19:46:50.755 INFO:teuthology.orchestra.run.smithi059.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-16T19:46:50.757 INFO:teuthology.orchestra.run.smithi059.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-16T19:46:50.758 INFO:teuthology.orchestra.run.smithi059.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-16T19:46:50.758 INFO:teuthology.orchestra.run.smithi059.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-16T19:46:50.758 INFO:teuthology.orchestra.run.smithi059.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-16T19:46:50.762 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-16T19:46:50.762 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-16T19:46:50.766 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-16T19:46:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-16T19:46:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-16T19:46:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-16T19:46:50.777 INFO:teuthology.orchestra.run.smithi059.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-16T19:46:50.777 INFO:teuthology.orchestra.run.smithi059.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-16T19:46:50.781 INFO:teuthology.orchestra.run.smithi059.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-16T19:46:50.782 INFO:teuthology.orchestra.run.smithi059.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-16T19:46:50.783 INFO:teuthology.orchestra.run.smithi059.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-16T19:46:50.784 INFO:teuthology.orchestra.run.smithi059.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-16T19:46:50.786 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-16T19:46:50.787 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-16T19:46:50.787 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-16T19:46:50.789 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-16T19:46:50.796 INFO:teuthology.orchestra.run.smithi059.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-16T19:46:50.799 INFO:teuthology.orchestra.run.smithi059.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-16T19:46:50.799 INFO:teuthology.orchestra.run.smithi059.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-16T19:46:50.800 INFO:teuthology.orchestra.run.smithi059.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-16T19:46:50.807 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-16T19:46:50.807 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-16T19:46:50.814 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-16T19:46:50.815 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-16T19:46:50.816 INFO:teuthology.orchestra.run.smithi059.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-16T19:46:50.816 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-16T19:46:50.816 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-16T19:46:50.821 INFO:teuthology.orchestra.run.smithi059.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-16T19:46:50.821 INFO:teuthology.orchestra.run.smithi059.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-16T19:46:50.826 INFO:teuthology.orchestra.run.smithi059.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-16T19:46:50.826 INFO:teuthology.orchestra.run.smithi059.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-16T19:46:50.827 INFO:teuthology.orchestra.run.smithi032.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-16T19:46:50.828 INFO:teuthology.orchestra.run.smithi059.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-16T19:46:50.830 INFO:teuthology.orchestra.run.smithi032.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-16T19:46:50.831 INFO:teuthology.orchestra.run.smithi032.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-16T19:46:50.833 INFO:teuthology.orchestra.run.smithi032.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-16T19:46:50.835 INFO:teuthology.orchestra.run.smithi059.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-16T19:46:50.838 INFO:teuthology.orchestra.run.smithi059.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-16T19:46:50.839 INFO:teuthology.orchestra.run.smithi059.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-16T19:46:50.848 INFO:teuthology.orchestra.run.smithi032.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-16T19:46:50.851 INFO:teuthology.orchestra.run.smithi059.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-16T19:46:50.851 INFO:teuthology.orchestra.run.smithi032.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-16T19:46:50.851 INFO:teuthology.orchestra.run.smithi032.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-16T19:46:50.853 INFO:teuthology.orchestra.run.smithi032.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-16T19:46:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-16T19:46:50.860 INFO:teuthology.orchestra.run.smithi059.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-16T19:46:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-16T19:46:50.864 INFO:teuthology.orchestra.run.smithi059.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-16T19:46:50.865 INFO:teuthology.orchestra.run.smithi059.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-16T19:46:50.920 INFO:teuthology.orchestra.run.smithi032.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-16T19:46:50.921 INFO:teuthology.orchestra.run.smithi032.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-16T19:46:50.923 INFO:teuthology.orchestra.run.smithi032.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-16T19:46:50.923 INFO:teuthology.orchestra.run.smithi032.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-16T19:46:50.924 INFO:teuthology.orchestra.run.smithi032.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-16T19:46:50.942 INFO:teuthology.orchestra.run.smithi032.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-16T19:46:50.944 INFO:teuthology.orchestra.run.smithi032.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-16T19:46:50.944 INFO:teuthology.orchestra.run.smithi032.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-16T19:46:50.958 INFO:teuthology.orchestra.run.smithi032.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-16T19:46:50.959 INFO:teuthology.orchestra.run.smithi032.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-16T19:46:50.966 INFO:teuthology.orchestra.run.smithi032.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-16T19:46:50.966 INFO:teuthology.orchestra.run.smithi032.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-16T19:46:50.967 INFO:teuthology.orchestra.run.smithi032.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-16T19:46:50.967 INFO:teuthology.orchestra.run.smithi032.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-16T19:46:50.968 INFO:teuthology.orchestra.run.smithi032.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-16T19:46:50.987 INFO:teuthology.orchestra.run.smithi032.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-16T19:46:50.990 INFO:teuthology.orchestra.run.smithi032.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-16T19:46:50.992 INFO:teuthology.orchestra.run.smithi032.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-16T19:46:50.993 INFO:teuthology.orchestra.run.smithi032.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-16T19:46:50.993 INFO:teuthology.orchestra.run.smithi032.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-16T19:46:51.007 INFO:teuthology.orchestra.run.smithi032.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-16T19:46:51.010 INFO:teuthology.orchestra.run.smithi032.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-16T19:46:51.010 INFO:teuthology.orchestra.run.smithi032.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-16T19:46:51.011 INFO:teuthology.orchestra.run.smithi032.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-16T19:46:51.011 INFO:teuthology.orchestra.run.smithi032.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-16T19:46:51.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-16T19:46:51.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-16T19:46:51.031 INFO:teuthology.orchestra.run.smithi032.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-16T19:46:51.032 INFO:teuthology.orchestra.run.smithi032.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-16T19:46:51.032 INFO:teuthology.orchestra.run.smithi032.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-16T19:46:51.048 INFO:teuthology.orchestra.run.smithi032.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-16T19:46:51.050 INFO:teuthology.orchestra.run.smithi032.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-16T19:46:51.050 INFO:teuthology.orchestra.run.smithi032.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-16T19:46:51.057 INFO:teuthology.orchestra.run.smithi032.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-16T19:46:51.057 INFO:teuthology.orchestra.run.smithi032.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-16T19:46:51.069 INFO:teuthology.orchestra.run.smithi032.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-16T19:46:51.071 INFO:teuthology.orchestra.run.smithi032.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-16T19:46:51.071 INFO:teuthology.orchestra.run.smithi032.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-16T19:46:51.515 INFO:teuthology.orchestra.run.smithi059.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1681-gaba3c1d2-1jammy [3,211 kB] 2024-03-16T19:46:52.095 INFO:teuthology.orchestra.run.smithi059.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1681-gaba3c1d2-1jammy [709 kB] 2024-03-16T19:46:52.228 INFO:teuthology.orchestra.run.smithi059.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1681-gaba3c1d2-1jammy [356 kB] 2024-03-16T19:46:52.290 INFO:teuthology.orchestra.run.smithi059.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1681-gaba3c1d2-1jammy [32.8 kB] 2024-03-16T19:46:52.327 INFO:teuthology.orchestra.run.smithi032.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1681-gaba3c1d2-1jammy [3,211 kB] 2024-03-16T19:46:52.346 INFO:teuthology.orchestra.run.smithi059.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1681-gaba3c1d2-1jammy [180 kB] 2024-03-16T19:46:52.354 INFO:teuthology.orchestra.run.smithi059.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1681-gaba3c1d2-1jammy [64.8 kB] 2024-03-16T19:46:52.356 INFO:teuthology.orchestra.run.smithi059.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1681-gaba3c1d2-1jammy [330 kB] 2024-03-16T19:46:52.424 INFO:teuthology.orchestra.run.smithi059.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1681-gaba3c1d2-1jammy [6,570 kB] 2024-03-16T19:46:53.041 INFO:teuthology.orchestra.run.smithi032.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1681-gaba3c1d2-1jammy [709 kB] 2024-03-16T19:46:53.126 INFO:teuthology.orchestra.run.smithi032.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1681-gaba3c1d2-1jammy [356 kB] 2024-03-16T19:46:53.201 INFO:teuthology.orchestra.run.smithi032.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1681-gaba3c1d2-1jammy [32.8 kB] 2024-03-16T19:46:53.202 INFO:teuthology.orchestra.run.smithi032.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1681-gaba3c1d2-1jammy [180 kB] 2024-03-16T19:46:53.269 INFO:teuthology.orchestra.run.smithi032.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1681-gaba3c1d2-1jammy [64.8 kB] 2024-03-16T19:46:53.271 INFO:teuthology.orchestra.run.smithi032.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1681-gaba3c1d2-1jammy [330 kB] 2024-03-16T19:46:53.284 INFO:teuthology.orchestra.run.smithi032.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1681-gaba3c1d2-1jammy [6,570 kB] 2024-03-16T19:46:53.670 INFO:teuthology.orchestra.run.smithi059.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1681-gaba3c1d2-1jammy [112 kB] 2024-03-16T19:46:53.720 INFO:teuthology.orchestra.run.smithi059.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1681-gaba3c1d2-1jammy [465 kB] 2024-03-16T19:46:53.783 INFO:teuthology.orchestra.run.smithi059.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1681-gaba3c1d2-1jammy [24.9 MB] 2024-03-16T19:46:54.325 INFO:teuthology.orchestra.run.smithi032.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1681-gaba3c1d2-1jammy [112 kB] 2024-03-16T19:46:54.331 INFO:teuthology.orchestra.run.smithi032.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1681-gaba3c1d2-1jammy [465 kB] 2024-03-16T19:46:54.398 INFO:teuthology.orchestra.run.smithi032.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1681-gaba3c1d2-1jammy [24.9 MB] 2024-03-16T19:46:57.967 INFO:teuthology.orchestra.run.smithi059.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1681-gaba3c1d2-1jammy [5,074 kB] 2024-03-16T19:46:57.975 INFO:teuthology.orchestra.run.smithi032.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1681-gaba3c1d2-1jammy [5,074 kB] 2024-03-16T19:46:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1681-gaba3c1d2-1jammy [244 kB] 2024-03-16T19:46:58.634 INFO:teuthology.orchestra.run.smithi032.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1681-gaba3c1d2-1jammy [124 kB] 2024-03-16T19:46:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1681-gaba3c1d2-1jammy [1,522 kB] 2024-03-16T19:46:58.748 INFO:teuthology.orchestra.run.smithi059.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1681-gaba3c1d2-1jammy [244 kB] 2024-03-16T19:46:58.816 INFO:teuthology.orchestra.run.smithi059.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1681-gaba3c1d2-1jammy [124 kB] 2024-03-16T19:46:58.820 INFO:teuthology.orchestra.run.smithi059.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1681-gaba3c1d2-1jammy [1,522 kB] 2024-03-16T19:46:58.858 INFO:teuthology.orchestra.run.smithi032.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1681-gaba3c1d2-1jammy [6,243 kB] 2024-03-16T19:46:59.113 INFO:teuthology.orchestra.run.smithi059.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1681-gaba3c1d2-1jammy [6,243 kB] 2024-03-16T19:46:59.690 INFO:teuthology.orchestra.run.smithi032.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1681-gaba3c1d2-1jammy [22.7 MB] 2024-03-16T19:47:00.173 INFO:teuthology.orchestra.run.smithi059.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1681-gaba3c1d2-1jammy [22.7 MB] 2024-03-16T19:47:02.904 INFO:teuthology.orchestra.run.smithi032.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1681-gaba3c1d2-1jammy [14.0 kB] 2024-03-16T19:47:02.904 INFO:teuthology.orchestra.run.smithi032.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1681-gaba3c1d2-1jammy [867 kB] 2024-03-16T19:47:02.974 INFO:teuthology.orchestra.run.smithi032.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1681-gaba3c1d2-1jammy [2,378 kB] 2024-03-16T19:47:03.294 INFO:teuthology.orchestra.run.smithi032.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1681-gaba3c1d2-1jammy [776 kB] 2024-03-16T19:47:03.400 INFO:teuthology.orchestra.run.smithi032.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1681-gaba3c1d2-1jammy [144 kB] 2024-03-16T19:47:03.404 INFO:teuthology.orchestra.run.smithi032.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1681-gaba3c1d2-1jammy [1,921 kB] 2024-03-16T19:47:03.726 INFO:teuthology.orchestra.run.smithi032.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1681-gaba3c1d2-1jammy [8,625 kB] 2024-03-16T19:47:04.006 INFO:teuthology.orchestra.run.smithi059.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1681-gaba3c1d2-1jammy [14.0 kB] 2024-03-16T19:47:04.007 INFO:teuthology.orchestra.run.smithi059.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1681-gaba3c1d2-1jammy [867 kB] 2024-03-16T19:47:04.024 INFO:teuthology.orchestra.run.smithi059.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1681-gaba3c1d2-1jammy [2,378 kB] 2024-03-16T19:47:04.386 INFO:teuthology.orchestra.run.smithi059.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1681-gaba3c1d2-1jammy [776 kB] 2024-03-16T19:47:04.504 INFO:teuthology.orchestra.run.smithi059.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1681-gaba3c1d2-1jammy [144 kB] 2024-03-16T19:47:04.567 INFO:teuthology.orchestra.run.smithi059.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1681-gaba3c1d2-1jammy [1,921 kB] 2024-03-16T19:47:04.867 INFO:teuthology.orchestra.run.smithi059.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1681-gaba3c1d2-1jammy [8,625 kB] 2024-03-16T19:47:05.073 INFO:teuthology.orchestra.run.smithi032.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1681-gaba3c1d2-1jammy [14.2 kB] 2024-03-16T19:47:05.074 INFO:teuthology.orchestra.run.smithi032.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1681-gaba3c1d2-1jammy [49.9 MB] 2024-03-16T19:47:06.248 INFO:teuthology.orchestra.run.smithi059.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1681-gaba3c1d2-1jammy [14.2 kB] 2024-03-16T19:47:06.249 INFO:teuthology.orchestra.run.smithi059.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1681-gaba3c1d2-1jammy [49.9 MB] 2024-03-16T19:47:09.349 INFO:teuthology.orchestra.run.smithi032.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1681-gaba3c1d2-1jammy [121 kB] 2024-03-16T19:47:09.351 INFO:teuthology.orchestra.run.smithi032.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1681-gaba3c1d2-1jammy [226 kB] 2024-03-16T19:47:09.356 INFO:teuthology.orchestra.run.smithi032.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1681-gaba3c1d2-1jammy [33.9 kB] 2024-03-16T19:47:09.357 INFO:teuthology.orchestra.run.smithi032.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1681-gaba3c1d2-1jammy [23.4 kB] 2024-03-16T19:47:09.358 INFO:teuthology.orchestra.run.smithi032.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1681-gaba3c1d2-1jammy [38.5 kB] 2024-03-16T19:47:09.360 INFO:teuthology.orchestra.run.smithi032.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1681-gaba3c1d2-1jammy [13.0 MB] 2024-03-16T19:47:10.024 INFO:teuthology.orchestra.run.smithi032.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1681-gaba3c1d2-1jammy [91.7 kB] 2024-03-16T19:47:10.026 INFO:teuthology.orchestra.run.smithi032.stdout:Get:125 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-1681-gaba3c1d2-1jammy [3,122 kB] 2024-03-16T19:47:10.161 INFO:teuthology.orchestra.run.smithi032.stdout:Get:126 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-1681-gaba3c1d2-1jammy [179 kB] 2024-03-16T19:47:10.178 INFO:teuthology.orchestra.run.smithi032.stdout:Get:127 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [744 MB] 2024-03-16T19:47:10.715 INFO:teuthology.orchestra.run.smithi059.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1681-gaba3c1d2-1jammy [121 kB] 2024-03-16T19:47:10.718 INFO:teuthology.orchestra.run.smithi059.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1681-gaba3c1d2-1jammy [226 kB] 2024-03-16T19:47:10.723 INFO:teuthology.orchestra.run.smithi059.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1681-gaba3c1d2-1jammy [33.9 kB] 2024-03-16T19:47:10.724 INFO:teuthology.orchestra.run.smithi059.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1681-gaba3c1d2-1jammy [23.4 kB] 2024-03-16T19:47:10.725 INFO:teuthology.orchestra.run.smithi059.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1681-gaba3c1d2-1jammy [38.5 kB] 2024-03-16T19:47:10.726 INFO:teuthology.orchestra.run.smithi059.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1681-gaba3c1d2-1jammy [13.0 MB] 2024-03-16T19:47:11.496 INFO:teuthology.orchestra.run.smithi059.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1681-gaba3c1d2-1jammy [91.7 kB] 2024-03-16T19:47:11.498 INFO:teuthology.orchestra.run.smithi059.stdout:Get:125 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-1681-gaba3c1d2-1jammy [3,122 kB] 2024-03-16T19:47:11.731 INFO:teuthology.orchestra.run.smithi059.stdout:Get:126 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-1681-gaba3c1d2-1jammy [179 kB] 2024-03-16T19:47:11.737 INFO:teuthology.orchestra.run.smithi059.stdout:Get:127 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [744 MB] 2024-03-16T19:47:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:Get:128 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [18.1 MB] 2024-03-16T19:47:52.430 INFO:teuthology.orchestra.run.smithi032.stdout:Get:129 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [83.1 MB] 2024-03-16T19:47:58.412 INFO:teuthology.orchestra.run.smithi032.stdout:Get:130 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [41.5 MB] 2024-03-16T19:48:00.837 INFO:teuthology.orchestra.run.smithi032.stdout:Get:131 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [158 MB] 2024-03-16T19:48:10.297 INFO:teuthology.orchestra.run.smithi032.stdout:Get:132 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [537 MB] 2024-03-16T19:48:25.343 INFO:teuthology.orchestra.run.smithi059.stdout:Get:128 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [18.1 MB] 2024-03-16T19:48:26.954 INFO:teuthology.orchestra.run.smithi059.stdout:Get:129 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [83.1 MB] 2024-03-16T19:48:39.630 INFO:teuthology.orchestra.run.smithi059.stdout:Get:130 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [41.5 MB] 2024-03-16T19:48:41.440 INFO:teuthology.orchestra.run.smithi059.stdout:Get:131 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [158 MB] 2024-03-16T19:48:42.781 INFO:teuthology.orchestra.run.smithi032.stdout:Get:133 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [1,179 MB] 2024-03-16T19:48:53.034 INFO:teuthology.orchestra.run.smithi059.stdout:Get:132 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [537 MB] 2024-03-16T19:49:38.106 INFO:teuthology.orchestra.run.smithi032.stdout:Get:134 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [16.4 MB] 2024-03-16T19:49:38.676 INFO:teuthology.orchestra.run.smithi032.stdout:Get:135 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librados2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [81.7 MB] 2024-03-16T19:49:42.330 INFO:teuthology.orchestra.run.smithi032.stdout:Get:136 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [11.2 MB] 2024-03-16T19:49:42.620 INFO:teuthology.orchestra.run.smithi032.stdout:Get:137 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [103 MB] 2024-03-16T19:49:46.956 INFO:teuthology.orchestra.run.smithi032.stdout:Get:138 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [173 MB] 2024-03-16T19:49:55.339 INFO:teuthology.orchestra.run.smithi032.stdout:Get:139 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [339 MB] 2024-03-16T19:50:10.101 INFO:teuthology.orchestra.run.smithi032.stdout:Get:140 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [1,696 kB] 2024-03-16T19:50:10.133 INFO:teuthology.orchestra.run.smithi032.stdout:Get:141 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [97.1 MB] 2024-03-16T19:50:15.148 INFO:teuthology.orchestra.run.smithi032.stdout:Get:142 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [3,003 kB] 2024-03-16T19:50:15.851 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 3,767 MB in 3min 25s (18.4 MB/s) 2024-03-16T19:50:15.993 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-16T19:50:16.095 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-16T19:50:16.101 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-16T19:50:16.150 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-16T19:50:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-16T19:50:16.421 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-16T19:50:16.452 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-16T19:50:16.735 INFO:teuthology.orchestra.run.smithi059.stdout:Get:133 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [1,179 MB] 2024-03-16T19:50:16.755 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cmd2. 2024-03-16T19:50:16.782 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-16T19:50:16.822 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-16T19:50:17.208 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-16T19:50:17.235 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-16T19:50:17.317 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-16T19:50:17.803 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-16T19:50:17.830 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-16T19:50:17.870 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-16T19:50:18.273 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-16T19:50:18.299 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-16T19:50:18.356 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-16T19:50:19.085 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-16T19:50:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:50:19.169 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:50:19.570 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-16T19:50:19.597 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:50:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:50:20.015 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-16T19:50:20.042 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:50:20.090 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:50:20.518 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-16T19:50:20.545 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-16T19:50:20.601 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-16T19:50:20.946 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librados2. 2024-03-16T19:50:20.973 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../010-librados2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:21.012 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librados2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:21.498 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librbd1. 2024-03-16T19:50:21.525 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../011-librbd1_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:21.565 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librbd1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:22.026 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs2. 2024-03-16T19:50:22.053 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:22.093 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:22.361 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rados. 2024-03-16T19:50:22.377 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:22.411 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rados (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:22.723 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-16T19:50:22.749 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:22.830 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-argparse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:23.108 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cephfs. 2024-03-16T19:50:23.135 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:23.175 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cephfs (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:23.494 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-16T19:50:23.521 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:23.560 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:23.888 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-prettytable. 2024-03-16T19:50:23.915 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-16T19:50:23.955 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-16T19:50:24.190 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rbd. 2024-03-16T19:50:24.206 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:24.240 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:24.635 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-16T19:50:24.662 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-16T19:50:24.710 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-16T19:50:25.054 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-16T19:50:25.070 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-16T19:50:25.129 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-16T19:50:25.490 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-16T19:50:25.517 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-16T19:50:25.557 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-16T19:50:25.912 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua5.1. 2024-03-16T19:50:25.938 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-16T19:50:25.987 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-16T19:50:26.348 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-any. 2024-03-16T19:50:26.375 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-16T19:50:26.423 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-16T19:50:26.700 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package zip. 2024-03-16T19:50:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-16T19:50:26.767 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking zip (3.0-12build2) ... 2024-03-16T19:50:27.087 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package unzip. 2024-03-16T19:50:27.114 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-16T19:50:27.153 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-16T19:50:27.582 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package luarocks. 2024-03-16T19:50:27.608 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-16T19:50:27.657 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-16T19:50:28.106 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librgw2. 2024-03-16T19:50:28.133 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../027-librgw2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:28.181 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librgw2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:28.758 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rgw. 2024-03-16T19:50:28.785 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:28.824 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:29.203 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-16T19:50:29.229 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-16T19:50:29.269 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-16T19:50:29.580 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libradosstriper1. 2024-03-16T19:50:29.597 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:29.638 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libradosstriper1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:30.000 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-common. 2024-03-16T19:50:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:30.058 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:31.538 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-base. 2024-03-16T19:50:31.565 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:31.615 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-base (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-16T19:50:32.228 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-16T19:50:32.267 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-16T19:50:32.653 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cheroot. 2024-03-16T19:50:32.680 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-16T19:50:32.728 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-16T19:50:33.114 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-16T19:50:33.142 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-16T19:50:33.198 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-16T19:50:33.534 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-16T19:50:33.561 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-16T19:50:33.609 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-16T19:50:33.920 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-16T19:50:33.947 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-16T19:50:33.986 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-16T19:50:34.364 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempora. 2024-03-16T19:50:34.392 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-16T19:50:34.439 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-16T19:50:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-portend. 2024-03-16T19:50:34.836 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-16T19:50:34.886 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-16T19:50:35.245 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-16T19:50:35.271 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-16T19:50:35.311 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-16T19:50:35.631 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-16T19:50:35.658 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-16T19:50:35.697 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-16T19:50:36.034 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-natsort. 2024-03-16T19:50:36.061 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-16T19:50:36.117 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-16T19:50:36.395 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-logutils. 2024-03-16T19:50:36.422 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-16T19:50:36.453 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-16T19:50:36.756 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-16T19:50:36.783 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-16T19:50:36.822 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-16T19:50:37.125 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-mako. 2024-03-16T19:50:37.152 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-16T19:50:37.192 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-16T19:50:37.520 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-16T19:50:37.547 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-16T19:50:37.586 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-16T19:50:37.848 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-16T19:50:37.875 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-16T19:50:37.906 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-16T19:50:38.192 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webob. 2024-03-16T19:50:38.219 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-16T19:50:38.258 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-16T19:50:38.628 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-waitress. 2024-03-16T19:50:38.655 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-16T19:50:38.698 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-16T19:50:39.006 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempita. 2024-03-16T19:50:39.033 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-16T19:50:39.072 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-16T19:50:39.378 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-paste. 2024-03-16T19:50:39.402 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-16T19:50:39.441 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-16T19:50:39.803 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-16T19:50:39.819 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-16T19:50:39.861 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-16T19:50:40.164 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-16T19:50:40.194 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-16T19:50:40.230 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-16T19:50:40.500 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webtest. 2024-03-16T19:50:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-16T19:50:40.558 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-16T19:50:40.794 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pecan. 2024-03-16T19:50:40.807 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-16T19:50:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-16T19:50:41.191 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-certifi. 2024-03-16T19:50:41.218 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-16T19:50:41.257 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-16T19:50:41.585 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-idna. 2024-03-16T19:50:41.612 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-16T19:50:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-16T19:50:41.979 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-urllib3. 2024-03-16T19:50:42.006 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-16T19:50:42.045 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-16T19:50:42.374 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests. 2024-03-16T19:50:42.401 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-16T19:50:42.440 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-16T19:50:42.735 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-16T19:50:42.762 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-16T19:50:42.801 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-16T19:50:43.113 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-16T19:50:43.140 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:43.609 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-16T19:50:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:43.675 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:44.003 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-16T19:50:44.030 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-16T19:50:44.069 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-16T19:50:44.381 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-openssl. 2024-03-16T19:50:44.408 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-16T19:50:44.447 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-16T19:50:44.742 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr. 2024-03-16T19:50:44.769 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:44.800 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:45.212 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mon. 2024-03-16T19:50:45.241 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:45.277 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mon (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:45.889 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-osd. 2024-03-16T19:50:45.917 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:45.956 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-osd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:47.134 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph. 2024-03-16T19:50:47.150 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../068-ceph_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:47.184 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:47.453 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-fuse. 2024-03-16T19:50:47.480 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:47.519 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:47.906 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mds. 2024-03-16T19:50:47.933 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:47.972 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mds (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:48.400 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephadm. 2024-03-16T19:50:48.427 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../071-cephadm_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:50:48.466 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:48.736 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-16T19:50:48.763 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-16T19:50:48.794 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-16T19:50:49.182 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-16T19:50:49.209 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-16T19:50:49.397 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-16T19:50:50.460 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-babel. 2024-03-16T19:50:50.487 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-16T19:50:50.526 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-16T19:50:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jinja2. 2024-03-16T19:50:50.881 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-16T19:50:50.920 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-16T19:50:51.265 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-16T19:50:51.293 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:51.331 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:51.651 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-16T19:50:51.679 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-16T19:50:51.717 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-16T19:50:52.029 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-routes. 2024-03-16T19:50:52.057 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-16T19:50:52.095 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-16T19:50:52.423 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-16T19:50:52.451 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:52.489 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:52.926 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-16T19:50:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-16T19:50:52.976 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-16T19:50:53.446 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-joblib. 2024-03-16T19:50:53.474 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-16T19:50:53.512 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-16T19:50:53.874 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-16T19:50:53.901 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-16T19:50:54.059 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-16T19:50:54.329 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn. 2024-03-16T19:50:54.357 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-16T19:50:54.395 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-16T19:50:55.179 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-16T19:50:55.207 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:55.245 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:56.174 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cachetools. 2024-03-16T19:50:56.201 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-16T19:50:56.240 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-16T19:50:56.543 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-16T19:50:56.570 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-16T19:50:56.609 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-16T19:50:56.954 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-16T19:50:56.982 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-16T19:50:57.021 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-16T19:50:57.190 INFO:teuthology.orchestra.run.smithi059.stdout:Get:134 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [16.4 MB] 2024-03-16T19:50:57.332 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rsa. 2024-03-16T19:50:57.359 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-16T19:50:57.398 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-16T19:50:57.684 INFO:teuthology.orchestra.run.smithi059.stdout:Get:135 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librados2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [81.7 MB] 2024-03-16T19:50:57.735 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-google-auth. 2024-03-16T19:50:57.763 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-16T19:50:57.802 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-16T19:50:58.088 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-16T19:50:58.105 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-16T19:50:58.138 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-16T19:50:58.424 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-websocket. 2024-03-16T19:50:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-16T19:50:58.491 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-16T19:50:58.819 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-16T19:50:58.847 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-16T19:50:58.885 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-16T19:50:59.591 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-16T19:50:59.608 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:50:59.649 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:50:59.977 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-16T19:51:00.005 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-16T19:51:00.043 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-16T19:51:00.388 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-16T19:51:00.416 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-16T19:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-16T19:51:00.657 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package jq. 2024-03-16T19:51:00.673 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-16T19:51:00.690 INFO:teuthology.orchestra.run.smithi059.stdout:Get:136 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [11.2 MB] 2024-03-16T19:51:00.707 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-16T19:51:00.942 INFO:teuthology.orchestra.run.smithi059.stdout:Get:137 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [103 MB] 2024-03-16T19:51:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package socat. 2024-03-16T19:51:00.976 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-16T19:51:01.017 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-16T19:51:01.337 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package xmlstarlet. 2024-03-16T19:51:01.354 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-16T19:51:01.395 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-16T19:51:01.750 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-test. 2024-03-16T19:51:01.766 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:01.807 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-test (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:04.253 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-volume. 2024-03-16T19:51:04.281 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:51:04.311 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-volume (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:Get:138 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [173 MB] 2024-03-16T19:51:04.882 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-16T19:51:04.910 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephfs-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:05.259 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-colorama. 2024-03-16T19:51:05.288 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-16T19:51:05.326 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-16T19:51:05.570 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephfs-shell. 2024-03-16T19:51:05.587 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:51:05.628 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephfs-shell (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:05.866 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephfs-top. 2024-03-16T19:51:05.883 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:51:05.916 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephfs-top (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:06.135 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-16T19:51:06.163 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:06.201 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs-dev (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:06.462 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-16T19:51:06.479 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-16T19:51:06.512 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-16T19:51:06.850 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-16T19:51:06.878 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-16T19:51:06.917 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-16T19:51:07.296 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package nvme-cli. 2024-03-16T19:51:07.324 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-16T19:51:07.362 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-16T19:51:07.792 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package pkg-config. 2024-03-16T19:51:07.820 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-16T19:51:07.859 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-16T19:51:08.103 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-16T19:51:08.120 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-16T19:51:08.153 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-16T19:51:08.598 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-16T19:51:08.626 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-16T19:51:08.664 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-16T19:51:08.942 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastescript. 2024-03-16T19:51:08.970 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-16T19:51:09.009 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-16T19:51:09.337 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pluggy. 2024-03-16T19:51:09.365 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-16T19:51:09.403 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-16T19:51:09.715 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-psutil. 2024-03-16T19:51:09.743 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-16T19:51:09.781 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-16T19:51:10.109 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-py. 2024-03-16T19:51:10.137 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-16T19:51:10.176 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-16T19:51:10.454 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pygments. 2024-03-16T19:51:10.482 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-16T19:51:10.513 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-16T19:51:10.975 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-16T19:51:11.003 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-16T19:51:11.041 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-16T19:51:11.248 INFO:teuthology.orchestra.run.smithi059.stdout:Get:139 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [339 MB] 2024-03-16T19:51:11.319 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-toml. 2024-03-16T19:51:11.347 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-16T19:51:11.377 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-16T19:51:11.646 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pytest. 2024-03-16T19:51:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-16T19:51:11.712 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-16T19:51:12.066 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplejson. 2024-03-16T19:51:12.088 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-16T19:51:12.124 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-16T19:51:12.377 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-16T19:51:12.405 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-16T19:51:12.435 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-16T19:51:13.073 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package radosgw. 2024-03-16T19:51:13.101 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../122-radosgw_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:13.130 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking radosgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:14.059 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-fuse. 2024-03-16T19:51:14.084 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:14.470 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-mirror. 2024-03-16T19:51:14.498 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../124-rbd-mirror_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:14.536 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:14.930 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-nbd. 2024-03-16T19:51:14.947 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../125-rbd-nbd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:14.980 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-nbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:15.242 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-common-dbg. 2024-03-16T19:51:15.270 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../126-ceph-common-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:15.308 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-common-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:23.087 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-fuse-dbg. 2024-03-16T19:51:23.115 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../127-ceph-fuse-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:23.152 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:23.755 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mds-dbg. 2024-03-16T19:51:23.783 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../128-ceph-mds-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:23.821 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mds-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:24.017 INFO:teuthology.orchestra.run.smithi059.stdout:Get:140 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [1,696 kB] 2024-03-16T19:51:24.049 INFO:teuthology.orchestra.run.smithi059.stdout:Get:141 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [97.1 MB] 2024-03-16T19:51:24.857 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-dbg. 2024-03-16T19:51:24.874 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../129-ceph-mgr-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:24.915 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:25.601 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mon-dbg. 2024-03-16T19:51:25.630 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../130-ceph-mon-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mon-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:27.420 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-osd-dbg. 2024-03-16T19:51:27.437 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../131-ceph-osd-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:27.478 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-osd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:32.789 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-test-dbg. 2024-03-16T19:51:32.806 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../132-ceph-test-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:32.847 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-test-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:34.782 INFO:teuthology.orchestra.run.smithi059.stdout:Get:142 https://2.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd-dbg amd64 19.0.0-1681-gaba3c1d2-1jammy [3,003 kB] 2024-03-16T19:51:35.510 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 3,767 MB in 4min 45s (13.2 MB/s) 2024-03-16T19:51:35.653 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-16T19:51:35.748 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-16T19:51:35.754 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-16T19:51:35.793 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-16T19:51:36.271 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-16T19:51:36.298 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-16T19:51:36.346 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-16T19:51:36.707 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cmd2. 2024-03-16T19:51:36.734 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-16T19:51:36.774 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-16T19:51:37.152 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-16T19:51:37.179 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-16T19:51:37.260 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-16T19:51:37.605 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-16T19:51:37.632 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-16T19:51:37.672 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-16T19:51:38.016 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-16T19:51:38.043 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-16T19:51:38.083 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-16T19:51:38.461 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-16T19:51:38.488 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:51:38.536 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:51:38.939 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-16T19:51:38.966 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:51:39.006 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:51:39.350 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-16T19:51:39.378 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-16T19:51:39.417 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:51:39.737 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-16T19:51:39.764 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-16T19:51:39.803 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-16T19:51:40.123 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librados2. 2024-03-16T19:51:40.150 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../010-librados2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:40.189 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librados2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:40.634 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librbd1. 2024-03-16T19:51:40.661 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../011-librbd1_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:40.700 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librbd1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:41.128 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs2. 2024-03-16T19:51:41.155 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:41.186 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:41.455 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rados. 2024-03-16T19:51:41.471 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:41.513 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rados (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:41.833 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-16T19:51:41.860 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:51:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-argparse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:42.152 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cephfs. 2024-03-16T19:51:42.168 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:42.210 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cephfs (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:42.521 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-16T19:51:42.548 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:51:42.588 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:42.918 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-prettytable. 2024-03-16T19:51:42.945 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-16T19:51:42.984 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-16T19:51:43.270 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rbd. 2024-03-16T19:51:43.286 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:43.328 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:43.681 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-16T19:51:43.708 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-16T19:51:43.748 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-16T19:51:44.000 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-16T19:51:44.028 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-16T19:51:44.075 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-16T19:51:44.289 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs2-dbg. 2024-03-16T19:51:44.318 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../133-libcephfs2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:44.355 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:44.378 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-16T19:51:44.406 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-16T19:51:44.445 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-16T19:51:44.782 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua5.1. 2024-03-16T19:51:44.782 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librados2-dbg. 2024-03-16T19:51:44.799 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../134-librados2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:44.808 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-16T19:51:44.832 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librados2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:44.848 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-16T19:51:45.151 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-any. 2024-03-16T19:51:45.167 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-16T19:51:45.201 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-16T19:51:45.412 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package zip. 2024-03-16T19:51:45.428 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-16T19:51:45.470 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking zip (3.0-12build2) ... 2024-03-16T19:51:45.782 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package unzip. 2024-03-16T19:51:45.809 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-16T19:51:45.848 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-16T19:51:45.985 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libradosstriper1-dbg. 2024-03-16T19:51:46.014 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../135-libradosstriper1-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:46.051 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libradosstriper1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:46.244 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package luarocks. 2024-03-16T19:51:46.271 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-16T19:51:46.310 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-16T19:51:46.463 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librbd1-dbg. 2024-03-16T19:51:46.491 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../136-librbd1-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:46.537 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librbd1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:46.710 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librgw2. 2024-03-16T19:51:46.737 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../027-librgw2_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:46.768 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librgw2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:47.304 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rgw. 2024-03-16T19:51:47.330 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:47.370 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:47.723 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-16T19:51:47.750 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-16T19:51:47.789 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-16T19:51:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librgw2-dbg. 2024-03-16T19:51:47.849 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../137-librgw2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:47.890 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librgw2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:48.101 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libradosstriper1. 2024-03-16T19:51:48.128 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:48.176 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libradosstriper1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:48.571 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-common. 2024-03-16T19:51:48.598 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:48.637 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:49.851 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package radosgw-dbg. 2024-03-16T19:51:49.879 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../138-radosgw-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:49.925 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking radosgw-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:50.059 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-base. 2024-03-16T19:51:50.086 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:50.135 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-base (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:50.722 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-16T19:51:50.749 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-16T19:51:50.788 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-16T19:51:51.091 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cheroot. 2024-03-16T19:51:51.118 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-16T19:51:51.157 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-16T19:51:51.494 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-16T19:51:51.521 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-16T19:51:51.561 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-16T19:51:51.830 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-16T19:51:51.857 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-16T19:51:51.897 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-16T19:51:52.183 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-16T19:51:52.210 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-16T19:51:52.249 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-16T19:51:52.561 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempora. 2024-03-16T19:51:52.577 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-16T19:51:52.619 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-16T19:51:52.922 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-portend. 2024-03-16T19:51:52.949 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-16T19:51:52.988 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-16T19:51:53.283 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-16T19:51:53.310 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-16T19:51:53.349 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-16T19:51:53.370 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-fuse-dbg. 2024-03-16T19:51:53.398 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../139-rbd-fuse-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:53.436 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:53.653 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-16T19:51:53.680 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-16T19:51:53.714 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-mirror-dbg. 2024-03-16T19:51:53.719 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-16T19:51:53.742 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../140-rbd-mirror-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:53.780 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-mirror-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:54.064 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-natsort. 2024-03-16T19:51:54.092 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-16T19:51:54.131 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-16T19:51:54.450 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-logutils. 2024-03-16T19:51:54.478 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-16T19:51:54.517 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-16T19:51:54.820 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-16T19:51:54.847 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-16T19:51:54.886 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-16T19:51:54.974 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-nbd-dbg. 2024-03-16T19:51:54.991 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../141-rbd-nbd-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:51:55.032 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-nbd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:55.148 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-mako. 2024-03-16T19:51:55.175 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-16T19:51:55.214 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-16T19:51:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-16T19:51:55.543 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-16T19:51:55.570 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-16T19:51:55.609 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-16T19:51:55.679 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-16T19:51:55.805 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-16T19:51:55.920 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-16T19:51:55.948 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-16T19:51:55.987 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-16T19:51:56.071 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-16T19:51:56.323 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webob. 2024-03-16T19:51:56.350 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-16T19:51:56.389 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-16T19:51:56.406 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-16T19:51:56.743 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-waitress. 2024-03-16T19:51:56.770 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-16T19:51:56.804 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-16T19:51:56.860 INFO:teuthology.orchestra.run.smithi032.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-16T19:51:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:51:57.079 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempita. 2024-03-16T19:51:57.106 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-16T19:51:57.145 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-16T19:51:57.448 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-paste. 2024-03-16T19:51:57.459 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user cephadm....done 2024-03-16T19:51:57.475 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-16T19:51:57.515 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-16T19:51:57.588 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-16T19:51:57.809 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-16T19:51:57.826 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-16T19:51:57.859 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-16T19:51:57.873 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-16T19:51:58.132 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-16T19:51:58.162 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-16T19:51:58.190 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-16T19:51:58.229 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-16T19:51:58.233 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-16T19:51:58.476 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-16T19:51:58.532 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webtest. 2024-03-16T19:51:58.559 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-16T19:51:58.598 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-16T19:51:58.735 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-16T19:51:58.862 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-16T19:51:58.910 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pecan. 2024-03-16T19:51:58.937 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-16T19:51:58.976 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-16T19:51:59.172 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-16T19:51:59.315 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-certifi. 2024-03-16T19:51:59.341 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-16T19:51:59.381 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-16T19:51:59.557 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-16T19:51:59.709 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-idna. 2024-03-16T19:51:59.736 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-16T19:51:59.775 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-16T19:51:59.833 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-16T19:52:00.084 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-16T19:52:00.103 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-urllib3. 2024-03-16T19:52:00.131 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-16T19:52:00.170 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-16T19:52:00.360 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-16T19:52:00.515 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests. 2024-03-16T19:52:00.542 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-16T19:52:00.581 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-16T19:52:00.604 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-16T19:52:00.859 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-16T19:52:00.871 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-16T19:52:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-16T19:52:00.917 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-16T19:52:01.139 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-16T19:52:01.212 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-16T19:52:01.228 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:01.270 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:01.407 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-argparse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:01.666 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-16T19:52:01.683 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-16T19:52:01.711 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:01.750 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:01.767 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-16T19:52:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-16T19:52:02.070 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-16T19:52:02.097 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-16T19:52:02.136 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-16T19:52:02.137 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-16T19:52:02.262 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-16T19:52:02.397 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-openssl. 2024-03-16T19:52:02.425 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-16T19:52:02.455 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-16T19:52:02.530 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-16T19:52:02.792 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr. 2024-03-16T19:52:02.819 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:02.858 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:02.893 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-16T19:52:02.920 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-16T19:52:02.983 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-16T19:52:03.084 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-16T19:52:03.245 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mon. 2024-03-16T19:52:03.272 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:03.311 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mon (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:03.435 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-16T19:52:03.570 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-16T19:52:03.846 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-16T19:52:04.040 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-osd. 2024-03-16T19:52:04.067 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:04.098 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-osd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:04.113 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-16T19:52:04.440 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up zip (3.0-12build2) ... 2024-03-16T19:52:04.557 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-16T19:52:05.134 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph. 2024-03-16T19:52:05.150 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../068-ceph_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:05.192 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:05.242 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-16T19:52:05.502 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-16T19:52:05.529 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-fuse. 2024-03-16T19:52:05.556 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:05.595 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:05.628 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-16T19:52:05.754 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-16T19:52:05.965 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mds. 2024-03-16T19:52:05.992 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:06.031 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mds (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:06.055 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-16T19:52:06.401 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephadm. 2024-03-16T19:52:06.428 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../071-cephadm_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:06.456 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-16T19:52:06.667 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:06.707 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-16T19:52:06.987 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-16T19:52:07.014 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-16T19:52:07.053 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-16T19:52:07.067 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-16T19:52:07.347 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-16T19:52:07.352 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-16T19:52:07.364 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-16T19:52:07.406 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-16T19:52:07.670 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-16T19:52:08.030 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-16T19:52:08.306 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-16T19:52:08.419 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-babel. 2024-03-16T19:52:08.432 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:08.448 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-16T19:52:08.494 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-16T19:52:08.749 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-idna (3.3-1) ... 2024-03-16T19:52:08.814 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jinja2. 2024-03-16T19:52:08.841 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-16T19:52:08.880 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-16T19:52:09.059 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-16T19:52:09.250 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-16T19:52:09.278 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:09.316 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:09.670 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-16T19:52:09.697 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-16T19:52:09.736 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-16T19:52:10.022 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-routes. 2024-03-16T19:52:10.050 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-16T19:52:10.089 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-16T19:52:10.198 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-16T19:52:10.417 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-16T19:52:10.433 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:52:10.433 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:10.492 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:10.575 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-16T19:52:10.826 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-16T19:52:10.945 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-16T19:52:10.952 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-16T19:52:10.973 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-16T19:52:11.012 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-16T19:52:11.079 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-16T19:52:11.380 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-16T19:52:11.482 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-joblib. 2024-03-16T19:52:11.498 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-16T19:52:11.540 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-16T19:52:11.648 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-16T19:52:11.835 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-16T19:52:11.863 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-16T19:52:11.907 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:52:11.987 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-16T19:52:12.008 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-16T19:52:12.298 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn. 2024-03-16T19:52:12.309 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-16T19:52:12.326 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-16T19:52:12.365 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-16T19:52:12.569 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-16T19:52:12.837 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-16T19:52:13.074 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-16T19:52:13.096 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-16T19:52:13.102 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:13.140 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:13.355 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-16T19:52:13.632 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-16T19:52:13.741 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-16T19:52:14.025 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-16T19:52:14.118 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cachetools. 2024-03-16T19:52:14.146 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-16T19:52:14.168 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-16T19:52:14.185 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-16T19:52:14.463 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-16T19:52:14.486 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-16T19:52:14.490 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-16T19:52:14.521 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-16T19:52:14.812 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-16T19:52:14.833 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-16T19:52:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-16T19:52:14.899 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-16T19:52:15.080 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-16T19:52:15.186 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rsa. 2024-03-16T19:52:15.206 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-16T19:52:15.213 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-16T19:52:15.252 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-16T19:52:15.332 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-16T19:52:15.421 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-16T19:52:15.573 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-google-auth. 2024-03-16T19:52:15.601 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-16T19:52:15.639 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-16T19:52:15.658 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-16T19:52:15.759 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-16T19:52:15.926 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-16T19:52:15.954 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-16T19:52:15.984 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-16T19:52:16.144 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-16T19:52:16.296 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-websocket. 2024-03-16T19:52:16.323 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-16T19:52:16.362 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-16T19:52:16.429 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-16T19:52:16.555 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-16T19:52:16.690 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-16T19:52:16.718 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-16T19:52:16.756 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-16T19:52:16.822 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:52:16.948 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-16T19:52:17.274 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-16T19:52:17.404 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-16T19:52:17.421 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:17.462 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:17.551 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-16T19:52:17.782 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-16T19:52:17.809 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-16T19:52:17.810 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-16T19:52:17.848 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-16T19:52:18.030 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-16T19:52:18.159 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-16T19:52:18.187 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-16T19:52:18.218 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-16T19:52:18.315 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-16T19:52:18.489 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package jq. 2024-03-16T19:52:18.506 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-16T19:52:18.539 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-16T19:52:18.583 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-16T19:52:18.775 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2024-03-16T19:52:18.803 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-16T19:52:18.825 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-16T19:52:18.841 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-16T19:52:19.161 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package xmlstarlet. 2024-03-16T19:52:19.178 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-16T19:52:19.194 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-16T19:52:19.219 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-16T19:52:19.496 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-16T19:52:19.574 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-test. 2024-03-16T19:52:19.590 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:19.623 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-test (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:19.848 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-16T19:52:19.966 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librados2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:20.067 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:20.168 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-16T19:52:20.496 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:20.622 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libradosstriper1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:20.748 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-16T19:52:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librados2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:21.125 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librbd1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:21.235 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:21.377 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-modules-core (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:21.503 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:21.763 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-16T19:52:21.764 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-16T19:52:21.978 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-volume. 2024-03-16T19:52:22.006 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:22.227 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-volume (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.229 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libradosstriper1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.355 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs-dev (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.481 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.607 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rados (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.649 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-16T19:52:22.677 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:22.708 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librgw2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.715 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephfs-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.809 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephfs-top (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:22.978 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-16T19:52:23.002 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-colorama. 2024-03-16T19:52:23.030 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-16T19:52:23.067 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-16T19:52:23.328 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephfs-shell. 2024-03-16T19:52:23.357 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:23.395 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephfs-shell (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:23.656 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephfs-top. 2024-03-16T19:52:23.684 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1681-gaba3c1d2-1jammy_all.deb ... 2024-03-16T19:52:23.722 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephfs-top (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:23.975 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-16T19:52:23.992 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:24.033 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs-dev (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.239 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librbd1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.311 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-16T19:52:24.339 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-16T19:52:24.365 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.369 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-16T19:52:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librgw2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.584 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.685 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-nbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.707 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-16T19:52:24.735 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-16T19:52:24.774 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-16T19:52:24.802 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cephfs (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:25.090 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-nbd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:25.144 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package nvme-cli. 2024-03-16T19:52:25.172 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-16T19:52:25.210 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-16T19:52:25.215 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephfs-shell (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:25.326 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:25.426 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:25.608 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package pkg-config. 2024-03-16T19:52:25.636 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-16T19:52:25.674 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-16T19:52:25.927 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-16T19:52:25.945 INFO:teuthology.orchestra.run.smithi032.stdout:Adding group ceph....done 2024-03-16T19:52:25.955 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-16T19:52:25.985 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-16T19:52:26.337 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user ceph....done 2024-03-16T19:52:26.371 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-16T19:52:26.400 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-16T19:52:26.438 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-16T19:52:26.480 INFO:teuthology.orchestra.run.smithi032.stdout:Setting system user ceph properties....done 2024-03-16T19:52:26.494 INFO:teuthology.orchestra.run.smithi032.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-16T19:52:26.639 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-16T19:52:26.741 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastescript. 2024-03-16T19:52:26.769 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-16T19:52:26.807 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-16T19:52:27.009 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-16T19:52:27.102 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pluggy. 2024-03-16T19:52:27.131 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-16T19:52:27.160 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-16T19:52:27.480 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-psutil. 2024-03-16T19:52:27.509 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-16T19:52:27.528 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephfs-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:27.546 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-16T19:52:27.745 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:52:27.745 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:52:27.869 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-py. 2024-03-16T19:52:27.896 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-16T19:52:27.936 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-16T19:52:28.220 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pygments. 2024-03-16T19:52:28.236 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-16T19:52:28.238 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-test (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:28.271 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-16T19:52:28.364 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up radosgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:28.733 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-16T19:52:28.761 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-16T19:52:28.799 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-16T19:52:28.888 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:52:28.888 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:52:29.109 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-toml. 2024-03-16T19:52:29.137 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-16T19:52:29.178 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-16T19:52:29.374 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-base (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:29.470 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pytest. 2024-03-16T19:52:29.498 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-16T19:52:29.536 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-16T19:52:29.720 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-16T19:52:29.882 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplejson. 2024-03-16T19:52:29.910 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-16T19:52:29.948 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-16T19:52:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-16T19:52:30.201 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-16T19:52:30.226 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mds (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:30.234 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-16T19:52:30.420 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-16T19:52:30.420 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-16T19:52:30.903 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-common-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:30.968 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package radosgw. 2024-03-16T19:52:30.996 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../122-radosgw_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:31.029 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:31.038 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking radosgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:31.253 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:52:31.254 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:52:31.747 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:31.900 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-fuse. 2024-03-16T19:52:31.927 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:31.966 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:31.975 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-16T19:52:31.976 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-16T19:52:32.294 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-mirror. 2024-03-16T19:52:32.322 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../124-rbd-mirror_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:32.360 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:32.483 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mds-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:32.601 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up radosgw-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:32.702 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-test-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:32.754 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-nbd. 2024-03-16T19:52:32.771 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../125-rbd-nbd_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:32.803 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-osd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:32.804 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-nbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:33.065 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-common-dbg. 2024-03-16T19:52:33.082 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../126-ceph-common-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:33.083 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-16T19:52:33.084 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-16T19:52:33.124 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-common-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:33.689 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-osd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:33.790 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-mirror-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:33.891 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:34.009 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:34.151 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mon (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:34.378 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-16T19:52:34.378 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-16T19:52:34.878 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:35.029 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:35.155 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-dashboard (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:35.306 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mon-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:35.441 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-volume (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:35.769 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-16T19:52:35.861 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-16T19:52:36.022 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-16T19:52:37.090 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-16T19:52:37.853 INFO:teuthology.orchestra.run.smithi032.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-16T19:52:37.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-16T19:52:37.914 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-03-16T19:52:38.114 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-03-16T19:52:38.115 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-03-16T19:52:38.299 INFO:teuthology.orchestra.run.smithi032.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:52:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-16T19:52:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-16T19:52:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-16T19:52:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-16T19:52:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-16T19:52:38.303 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-03-16T19:52:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-16T19:52:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-16T19:52:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: libengine-pkcs11-openssl 2024-03-16T19:52:38.344 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-03-16T19:52:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-16T19:52:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-16T19:52:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-16T19:52:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-16T19:52:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-16T19:52:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-16T19:52:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-16T19:52:38.347 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages will be upgraded: 2024-03-16T19:52:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: libssl3 2024-03-16T19:52:38.445 INFO:teuthology.orchestra.run.smithi032.stdout:1 upgraded, 39 newly installed, 0 to remove and 283 not upgraded. 2024-03-16T19:52:38.445 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 54.6 MB of archives. 2024-03-16T19:52:38.445 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-16T19:52:38.445 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-16T19:52:38.676 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-16T19:52:38.810 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-16T19:52:38.833 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-16T19:52:38.834 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-16T19:52:38.838 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-16T19:52:38.841 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-16T19:52:38.841 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-16T19:52:38.844 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-16T19:52:38.846 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-16T19:52:38.847 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-16T19:52:38.859 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-16T19:52:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-16T19:52:39.441 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-16T19:52:39.444 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-16T19:52:39.503 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-16T19:52:39.508 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-16T19:52:39.508 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-16T19:52:39.516 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-16T19:52:39.519 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-16T19:52:39.532 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-16T19:52:39.533 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-16T19:52:39.569 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-16T19:52:39.570 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-16T19:52:39.583 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-16T19:52:39.623 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-16T19:52:39.629 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-16T19:52:39.630 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-16T19:52:39.631 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-16T19:52:39.631 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-16T19:52:39.648 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-16T19:52:39.650 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-16T19:52:40.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-16T19:52:40.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-16T19:52:40.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-16T19:52:40.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-16T19:52:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-16T19:52:40.066 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-16T19:52:40.086 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-16T19:52:40.087 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-16T19:52:40.510 INFO:teuthology.orchestra.run.smithi032.stdout:Preconfiguring packages ... 2024-03-16T19:52:40.544 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-fuse-dbg. 2024-03-16T19:52:40.561 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../127-ceph-fuse-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:40.602 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:40.662 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 54.6 MB in 2s (31.5 MB/s) 2024-03-16T19:52:40.807 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package flex. 2024-03-16T19:52:40.908 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173947 files and directories currently installed.) 2024-03-16T19:52:40.914 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-16T19:52:40.972 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-16T19:52:41.171 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mds-dbg. 2024-03-16T19:52:41.188 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../128-ceph-mds-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:41.229 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mds-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:41.648 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-16T19:52:41.845 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-16T19:52:42.265 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-dbg. 2024-03-16T19:52:42.274 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:52:42.294 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../129-ceph-mgr-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:42.332 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:42.924 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package bison. 2024-03-16T19:52:43.001 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mon-dbg. 2024-03-16T19:52:43.017 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../130-ceph-mon-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:43.032 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174020 files and directories currently installed.) 2024-03-16T19:52:43.038 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-16T19:52:43.059 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mon-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:43.084 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-16T19:52:43.494 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-16T19:52:43.523 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-16T19:52:43.561 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-16T19:52:43.839 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package dump. 2024-03-16T19:52:43.856 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-16T19:52:43.889 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking dump (0.4b47-3) ... 2024-03-16T19:52:44.184 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package indent. 2024-03-16T19:52:44.201 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:44.243 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-16T19:52:44.512 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-16T19:52:44.528 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-16T19:52:44.570 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-16T19:52:44.803 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-osd-dbg. 2024-03-16T19:52:44.831 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../131-ceph-osd-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:44.869 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-osd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-16T19:52:44.953 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-16T19:52:44.991 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-16T19:52:45.277 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-16T19:52:45.305 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-16T19:52:45.343 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-16T19:52:45.755 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-16T19:52:45.784 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-16T19:52:45.830 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-16T19:52:46.125 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-16T19:52:46.153 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-16T19:52:46.191 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-16T19:52:46.456 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libjson-perl. 2024-03-16T19:52:46.482 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-16T19:52:46.519 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-16T19:52:46.881 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-16T19:52:46.909 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-16T19:52:46.955 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-16T19:52:48.709 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-16T19:52:48.737 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-16T19:52:48.775 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:52:49.037 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-16T19:52:49.054 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-16T19:52:49.095 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:52:49.448 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-16T19:52:49.476 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-16T19:52:49.526 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:52:49.943 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-16T19:52:49.971 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-16T19:52:50.025 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-16T19:52:50.031 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-test-dbg. 2024-03-16T19:52:50.052 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../132-ceph-test-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:52:50.114 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-test-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:52:50.370 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-16T19:52:50.399 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-16T19:52:50.437 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-16T19:52:50.831 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-16T19:52:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:50.915 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:52:51.326 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-16T19:52:51.355 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:52:51.713 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-16T19:52:51.743 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:51.779 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:52:52.032 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-16T19:52:52.049 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-16T19:52:52.089 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:52:52.602 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-16T19:52:52.635 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-16T19:52:52.660 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-16T19:52:53.033 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package modemmanager. 2024-03-16T19:52:53.061 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-16T19:52:53.099 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:52:53.705 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package network-manager. 2024-03-16T19:52:53.734 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-16T19:52:53.780 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-16T19:52:54.412 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ppp. 2024-03-16T19:52:54.441 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-16T19:52:54.710 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-16T19:52:55.042 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package pptp-linux. 2024-03-16T19:52:55.071 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-16T19:52:55.109 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-16T19:52:55.337 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-16T19:52:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-16T19:52:55.386 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-16T19:52:55.639 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-16T19:52:55.668 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-16T19:52:55.706 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-client-common (238) ... 2024-03-16T19:52:56.018 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-16T19:52:56.047 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:56.085 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:52:56.682 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-common. 2024-03-16T19:52:56.711 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-16T19:52:56.775 INFO:teuthology.orchestra.run.smithi032.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-16T19:52:56.805 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-common (238) ... 2024-03-16T19:52:57.306 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-14. 2024-03-16T19:52:57.335 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:52:57.375 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:52:59.361 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql. 2024-03-16T19:52:59.378 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-16T19:52:59.419 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql (14+238) ... 2024-03-16T19:52:59.697 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-client. 2024-03-16T19:52:59.725 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-16T19:52:59.771 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-client (14+238) ... 2024-03-16T19:53:00.066 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-16T19:53:00.094 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-16T19:53:00.140 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-16T19:53:00.460 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package tcl8.6. 2024-03-16T19:53:00.489 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-16T19:53:00.526 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-16T19:53:00.829 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package tcl. 2024-03-16T19:53:00.858 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-16T19:53:00.900 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-16T19:53:01.207 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package wpasupplicant. 2024-03-16T19:53:01.236 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-16T19:53:01.239 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs2-dbg. 2024-03-16T19:53:01.256 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../133-libcephfs2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:01.265 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-16T19:53:01.297 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:01.645 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-16T19:53:01.662 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-16T19:53:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-16T19:53:01.741 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librados2-dbg. 2024-03-16T19:53:01.770 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../134-librados2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:01.807 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librados2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:02.165 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-16T19:53:02.194 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-16T19:53:02.248 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-16T19:53:02.579 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-client-common (238) ... 2024-03-16T19:53:02.805 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-16T19:53:02.869 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libradosstriper1-dbg. 2024-03-16T19:53:02.885 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../135-libradosstriper1-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:02.926 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libradosstriper1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:02.931 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-16T19:53:03.057 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:53:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-16T19:53:03.259 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-16T19:53:03.304 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librbd1-dbg. 2024-03-16T19:53:03.321 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../136-librbd1-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:03.354 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librbd1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:03.377 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:53:03.934 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-16T19:53:04.025 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-16T19:53:04.384 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-16T19:53:04.384 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-16T19:53:04.565 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librgw2-dbg. 2024-03-16T19:53:04.582 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../137-librgw2-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:04.623 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librgw2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:04.895 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-16T19:53:05.047 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:53:05.172 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-16T19:53:05.298 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-16T19:53:05.425 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-16T19:53:05.526 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-16T19:53:05.635 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-16T19:53:06.476 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package radosgw-dbg. 2024-03-16T19:53:06.493 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../138-radosgw-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:06.534 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking radosgw-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:06.781 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-16T19:53:07.046 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up dump (0.4b47-3) ... 2024-03-16T19:53:07.221 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-16T19:53:07.331 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:53:07.507 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-16T19:53:07.616 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-16T19:53:07.742 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-16T19:53:07.868 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-16T19:53:08.017 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-16T19:53:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-16T19:53:08.213 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-16T19:53:08.372 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-16T19:53:08.473 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:53:08.583 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-16T19:53:09.135 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-client (14+238) ... 2024-03-16T19:53:09.269 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-16T19:53:09.765 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout:which means they are currently configured by ifupdown: 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout:- ens1f0 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout:remove their configuration from /etc/network/interfaces. 2024-03-16T19:53:09.766 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:09.787 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-fuse-dbg. 2024-03-16T19:53:09.816 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../139-rbd-fuse-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:09.845 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-16T19:53:10.106 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-mirror-dbg. 2024-03-16T19:53:10.134 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../140-rbd-mirror-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:10.180 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-mirror-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:10.221 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-16T19:53:10.541 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-16T19:53:11.038 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:53:11.181 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-16T19:53:11.324 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-16T19:53:11.393 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-nbd-dbg. 2024-03-16T19:53:11.420 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../141-rbd-nbd-dbg_19.0.0-1681-gaba3c1d2-1jammy_amd64.deb ... 2024-03-16T19:53:11.450 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-common (238) ... 2024-03-16T19:53:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-nbd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:11.798 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-16T19:53:12.055 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-16T19:53:12.181 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-16T19:53:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:Adding user postgres to group ssl-cert 2024-03-16T19:53:12.449 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-16T19:53:12.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:12.654 INFO:teuthology.orchestra.run.smithi032.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-16T19:53:12.775 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-16T19:53:12.862 INFO:teuthology.orchestra.run.smithi032.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-16T19:53:12.862 INFO:teuthology.orchestra.run.smithi032.stdout:Removing obsolete dictionary files: 2024-03-16T19:53:13.169 INFO:teuthology.orchestra.run.smithi059.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-16T19:53:13.233 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:13.441 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-16T19:53:13.769 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user cephadm....done 2024-03-16T19:53:13.845 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:53:13.914 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-16T19:53:14.168 INFO:teuthology.orchestra.run.smithi032.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-16T19:53:14.191 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-16T19:53:14.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:This user must also own the server process. 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:The database cluster will be initialized with locale "en_US". 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:The default text search configuration will be set to "english". 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:Data page checksums are disabled. 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:53:14.257 INFO:teuthology.orchestra.run.smithi032.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-16T19:53:14.258 INFO:teuthology.orchestra.run.smithi032.stdout:creating subdirectories ... ok 2024-03-16T19:53:14.258 INFO:teuthology.orchestra.run.smithi032.stdout:selecting dynamic shared memory implementation ... posix 2024-03-16T19:53:14.274 INFO:teuthology.orchestra.run.smithi032.stdout:selecting default max_connections ... 100 2024-03-16T19:53:14.297 INFO:teuthology.orchestra.run.smithi032.stdout:selecting default shared_buffers ... 128MB 2024-03-16T19:53:14.307 INFO:teuthology.orchestra.run.smithi032.stdout:selecting default time zone ... Etc/UTC 2024-03-16T19:53:14.308 INFO:teuthology.orchestra.run.smithi032.stdout:creating configuration files ... ok 2024-03-16T19:53:14.423 INFO:teuthology.orchestra.run.smithi032.stdout:running bootstrap script ... ok 2024-03-16T19:53:14.442 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-16T19:53:14.568 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-16T19:53:14.828 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-16T19:53:15.087 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-16T19:53:15.096 INFO:teuthology.orchestra.run.smithi032.stdout:performing post-bootstrap initialization ... ok 2024-03-16T19:53:15.213 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-16T19:53:15.506 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-16T19:53:15.722 INFO:teuthology.orchestra.run.smithi032.stdout:syncing data to disk ... ok 2024-03-16T19:53:15.875 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-16T19:53:16.126 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-16T19:53:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-16T19:53:16.402 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-16T19:53:16.687 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-16T19:53:16.980 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-16T19:53:17.256 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-16T19:53:17.515 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-16T19:53:17.942 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-argparse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:18.218 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-16T19:53:18.344 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-16T19:53:18.478 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-16T19:53:18.738 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-16T19:53:18.880 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-16T19:53:18.927 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:53:19.053 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-16T19:53:19.140 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-16T19:53:19.179 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up postgresql (14+238) ... 2024-03-16T19:53:19.305 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:53:19.406 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:53:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-16T19:53:19.629 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-16T19:53:19.660 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-16T19:53:19.660 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-16T19:53:19.693 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-16T19:53:19.819 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-16T19:53:20.195 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-16T19:53:20.321 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-16T19:53:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-16T19:53:20.597 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-16T19:53:20.661 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-16T19:53:20.804 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-16T19:53:20.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-16T19:53:21.250 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up zip (3.0-12build2) ... 2024-03-16T19:53:21.376 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-16T19:53:21.857 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-16T19:53:21.956 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-16T19:53:22.103 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-16T19:53:22.370 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-16T19:53:22.465 INFO:teuthology.orchestra.run.smithi032.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-16T19:53:22.470 DEBUG:teuthology.parallel:result is None 2024-03-16T19:53:22.496 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-16T19:53:22.623 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-16T19:53:22.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-16T19:53:23.316 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-16T19:53:23.584 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-16T19:53:24.003 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-16T19:53:24.304 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-16T19:53:24.647 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-16T19:53:25.015 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-16T19:53:25.266 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-16T19:53:25.392 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:25.685 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-idna (3.3-1) ... 2024-03-16T19:53:26.003 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-16T19:53:27.208 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-16T19:53:27.435 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:53:27.595 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-16T19:53:27.862 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-16T19:53:27.988 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-16T19:53:28.114 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-16T19:53:28.416 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-16T19:53:28.684 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-16T19:53:28.934 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:53:29.044 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-16T19:53:29.362 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-16T19:53:29.872 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-16T19:53:30.231 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-16T19:53:30.491 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-16T19:53:30.783 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-16T19:53:31.051 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-16T19:53:31.177 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-16T19:53:31.478 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-16T19:53:31.604 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-16T19:53:31.939 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-16T19:53:32.248 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-16T19:53:32.499 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-16T19:53:32.625 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-16T19:53:32.734 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-16T19:53:32.806 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-16T19:53:33.053 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-16T19:53:33.179 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-16T19:53:33.589 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-16T19:53:33.865 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-16T19:53:33.991 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-16T19:53:34.250 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-16T19:53:34.352 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-16T19:53:34.694 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-16T19:53:35.004 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-16T19:53:35.297 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-16T19:53:35.508 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-16T19:53:35.776 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-16T19:53:36.036 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-16T19:53:36.314 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-16T19:53:36.699 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-16T19:53:37.001 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-16T19:53:37.319 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-16T19:53:37.445 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librados2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:37.572 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:37.698 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-16T19:53:38.017 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:38.143 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libradosstriper1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-16T19:53:38.545 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librados2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:38.688 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librbd1 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:38.814 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:38.940 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-modules-core (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:39.066 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:39.291 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-16T19:53:39.291 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-16T19:53:39.759 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libradosstriper1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:39.894 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs-dev (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:40.037 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:40.180 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rados (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:40.657 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librgw2 (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:40.783 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephfs-top (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:40.884 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-16T19:53:42.145 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librbd1-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.280 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.406 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librgw2-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.532 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-fuse (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.650 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-nbd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.751 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:42.876 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cephfs (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:43.002 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-nbd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:43.128 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephfs-shell (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:43.280 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-fuse-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:43.414 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-common (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:43.915 INFO:teuthology.orchestra.run.smithi059.stdout:Adding group ceph....done 2024-03-16T19:53:44.275 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user ceph....done 2024-03-16T19:53:44.519 INFO:teuthology.orchestra.run.smithi059.stdout:Setting system user ceph properties....done 2024-03-16T19:53:44.545 INFO:teuthology.orchestra.run.smithi059.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-16T19:53:44.694 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-16T19:53:45.026 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-16T19:53:45.676 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephfs-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:45.896 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:53:45.896 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:53:46.295 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-test (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:46.421 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up radosgw (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:46.895 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:53:46.896 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:53:47.331 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-base (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:47.691 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-16T19:53:48.141 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mds (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:48.376 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-16T19:53:48.376 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-16T19:53:48.834 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-common-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:48.960 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-mirror (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:49.190 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:53:49.190 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:53:49.661 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:49.883 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-16T19:53:49.883 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-16T19:53:50.329 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mds-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:50.464 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up radosgw-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:50.682 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-test-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:50.808 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-osd (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:51.096 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-16T19:53:51.096 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-16T19:53:51.568 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:51.677 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-osd-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:51.803 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-mirror-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:51.929 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:52.056 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:52.207 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mon (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-16T19:53:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-16T19:53:52.916 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-cephadm (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:53.067 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:53.193 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-dashboard (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:53.346 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mon-dbg (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:53.453 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-volume (19.0.0-1681-gaba3c1d2-1jammy) ... 2024-03-16T19:53:53.806 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-16T19:53:53.915 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-16T19:53:54.059 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-16T19:53:55.293 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-16T19:53:55.959 INFO:teuthology.orchestra.run.smithi059.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-16T19:53:55.964 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-16T19:53:56.063 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-03-16T19:53:56.262 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-03-16T19:53:56.263 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-03-16T19:53:56.446 INFO:teuthology.orchestra.run.smithi059.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-16T19:53:56.447 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-16T19:53:56.448 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-16T19:53:56.448 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-16T19:53:56.448 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-03-16T19:53:56.448 INFO:teuthology.orchestra.run.smithi059.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-16T19:53:56.448 INFO:teuthology.orchestra.run.smithi059.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-16T19:53:56.449 INFO:teuthology.orchestra.run.smithi059.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-16T19:53:56.449 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-16T19:53:56.449 INFO:teuthology.orchestra.run.smithi059.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-16T19:53:56.450 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-03-16T19:53:56.450 INFO:teuthology.orchestra.run.smithi059.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-16T19:53:56.450 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-16T19:53:56.450 INFO:teuthology.orchestra.run.smithi059.stdout: libengine-pkcs11-openssl 2024-03-16T19:53:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-03-16T19:53:56.491 INFO:teuthology.orchestra.run.smithi059.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-16T19:53:56.491 INFO:teuthology.orchestra.run.smithi059.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-16T19:53:56.491 INFO:teuthology.orchestra.run.smithi059.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-16T19:53:56.491 INFO:teuthology.orchestra.run.smithi059.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-16T19:53:56.491 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-16T19:53:56.492 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-16T19:53:56.492 INFO:teuthology.orchestra.run.smithi059.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-16T19:53:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages will be upgraded: 2024-03-16T19:53:56.493 INFO:teuthology.orchestra.run.smithi059.stdout: libssl3 2024-03-16T19:53:56.565 INFO:teuthology.orchestra.run.smithi059.stdout:1 upgraded, 39 newly installed, 0 to remove and 283 not upgraded. 2024-03-16T19:53:56.565 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 54.6 MB of archives. 2024-03-16T19:53:56.565 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-16T19:53:56.565 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-16T19:53:56.868 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-16T19:53:57.105 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-16T19:53:57.128 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-16T19:53:57.130 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-16T19:53:57.135 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-16T19:53:57.138 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-16T19:53:57.140 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-16T19:53:57.142 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-16T19:53:57.144 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-16T19:53:57.145 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-16T19:53:57.150 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-16T19:53:57.182 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-16T19:53:57.862 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-16T19:53:57.867 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-16T19:53:57.867 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-16T19:53:57.874 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-16T19:53:57.874 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-16T19:53:57.885 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-16T19:53:57.889 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-16T19:53:57.906 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-16T19:53:57.907 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-16T19:53:57.941 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-16T19:53:57.944 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-16T19:53:57.972 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-16T19:53:58.022 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-16T19:53:58.030 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-16T19:53:58.031 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-16T19:53:58.032 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-16T19:53:58.033 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-16T19:53:58.058 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-16T19:53:58.061 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-16T19:53:58.459 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-16T19:53:58.459 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-16T19:53:58.459 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-16T19:53:58.459 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-16T19:53:58.460 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-16T19:53:58.460 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-16T19:53:58.480 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-16T19:53:58.481 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-16T19:53:58.908 INFO:teuthology.orchestra.run.smithi059.stdout:Preconfiguring packages ... 2024-03-16T19:53:59.079 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 54.6 MB in 2s (27.6 MB/s) 2024-03-16T19:53:59.200 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package flex. 2024-03-16T19:53:59.304 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173947 files and directories currently installed.) 2024-03-16T19:53:59.310 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-16T19:53:59.356 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-16T19:53:59.873 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-16T19:54:00.037 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-16T19:54:00.448 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:54:00.980 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package bison. 2024-03-16T19:54:01.086 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174020 files and directories currently installed.) 2024-03-16T19:54:01.092 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-16T19:54:01.132 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-16T19:54:01.541 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-16T19:54:01.570 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-16T19:54:01.608 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-16T19:54:01.945 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package dump. 2024-03-16T19:54:01.973 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-16T19:54:02.011 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking dump (0.4b47-3) ... 2024-03-16T19:54:02.314 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package indent. 2024-03-16T19:54:02.331 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:02.389 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-16T19:54:02.650 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-16T19:54:02.668 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-16T19:54:02.708 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-16T19:54:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-16T19:54:03.065 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-16T19:54:03.103 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-16T19:54:03.339 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-16T19:54:03.353 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-16T19:54:03.397 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-16T19:54:03.751 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-16T19:54:03.779 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-16T19:54:03.817 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-16T19:54:04.069 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-16T19:54:04.086 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-16T19:54:04.128 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-16T19:54:04.380 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjson-perl. 2024-03-16T19:54:04.397 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-16T19:54:04.439 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-16T19:54:04.717 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-16T19:54:04.745 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-16T19:54:04.792 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-16T19:54:06.477 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-16T19:54:06.506 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-16T19:54:06.544 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:54:06.838 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-16T19:54:06.855 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-16T19:54:06.896 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:54:07.174 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-16T19:54:07.203 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-16T19:54:07.232 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:54:07.577 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-16T19:54:07.605 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-16T19:54:07.643 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-16T19:54:07.971 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-16T19:54:08.000 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-16T19:54:08.038 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-16T19:54:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-16T19:54:08.419 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:08.449 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:08.777 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-16T19:54:08.805 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:08.843 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:54:09.145 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-16T19:54:09.162 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:09.204 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:54:09.448 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-16T19:54:09.476 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-16T19:54:09.514 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:54:10.060 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-16T19:54:10.088 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-16T19:54:10.126 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-16T19:54:10.471 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package modemmanager. 2024-03-16T19:54:10.499 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-16T19:54:10.537 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:54:11.160 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package network-manager. 2024-03-16T19:54:11.188 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-16T19:54:11.235 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-16T19:54:11.842 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ppp. 2024-03-16T19:54:11.870 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-16T19:54:12.139 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-16T19:54:12.438 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package pptp-linux. 2024-03-16T19:54:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-16T19:54:12.504 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-16T19:54:12.765 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-16T19:54:12.782 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-16T19:54:12.823 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-16T19:54:13.126 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-16T19:54:13.155 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-16T19:54:13.193 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-client-common (238) ... 2024-03-16T19:54:13.496 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-16T19:54:13.525 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:13.563 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:14.143 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-common. 2024-03-16T19:54:14.172 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-16T19:54:14.228 INFO:teuthology.orchestra.run.smithi059.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-16T19:54:14.257 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-common (238) ... 2024-03-16T19:54:14.725 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-14. 2024-03-16T19:54:14.754 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-16T19:54:14.795 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:16.061 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql. 2024-03-16T19:54:16.078 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-16T19:54:16.110 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql (14+238) ... 2024-03-16T19:54:16.813 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-client. 2024-03-16T19:54:16.830 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-16T19:54:16.871 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-client (14+238) ... 2024-03-16T19:54:17.114 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-16T19:54:17.132 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-16T19:54:17.181 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-16T19:54:17.500 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package tcl8.6. 2024-03-16T19:54:17.529 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-16T19:54:17.567 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-16T19:54:17.820 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package tcl. 2024-03-16T19:54:17.837 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-16T19:54:17.874 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-16T19:54:18.222 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package wpasupplicant. 2024-03-16T19:54:18.251 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-16T19:54:18.289 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-16T19:54:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-16T19:54:18.668 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-16T19:54:18.709 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-16T19:54:19.088 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-16T19:54:19.116 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-16T19:54:19.163 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-16T19:54:19.434 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-client-common (238) ... 2024-03-16T19:54:19.643 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-16T19:54:19.794 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-16T19:54:19.920 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:20.046 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-16T19:54:20.172 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-16T19:54:20.298 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:20.838 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-16T19:54:20.920 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-16T19:54:21.309 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-16T19:54:21.309 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-16T19:54:21.781 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-16T19:54:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-16T19:54:22.044 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-16T19:54:22.170 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-16T19:54:22.295 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-16T19:54:22.430 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-16T19:54:22.558 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-16T19:54:23.186 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-16T19:54:23.312 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up dump (0.4b47-3) ... 2024-03-16T19:54:23.422 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-16T19:54:23.488 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:54:23.614 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-16T19:54:23.740 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-16T19:54:23.866 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-16T19:54:23.984 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-16T19:54:24.074 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-16T19:54:24.177 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-16T19:54:24.311 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-16T19:54:24.487 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-16T19:54:24.622 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:54:24.748 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-16T19:54:24.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-client (14+238) ... 2024-03-16T19:54:25.041 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-16T19:54:25.547 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-16T19:54:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:which means they are currently configured by ifupdown: 2024-03-16T19:54:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:- ens1f0 2024-03-16T19:54:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-16T19:54:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:remove their configuration from /etc/network/interfaces. 2024-03-16T19:54:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:25.670 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-16T19:54:25.983 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-16T19:54:26.298 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-16T19:54:26.802 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-16T19:54:27.329 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-16T19:54:27.457 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-16T19:54:27.581 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-common (238) ... 2024-03-16T19:54:28.475 INFO:teuthology.orchestra.run.smithi059.stdout:Adding user postgres to group ssl-cert 2024-03-16T19:54:28.730 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:28.730 INFO:teuthology.orchestra.run.smithi059.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-16T19:54:28.917 INFO:teuthology.orchestra.run.smithi059.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-16T19:54:28.917 INFO:teuthology.orchestra.run.smithi059.stdout:Removing obsolete dictionary files: 2024-03-16T19:54:29.455 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-16T19:54:29.824 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-16T19:54:30.158 INFO:teuthology.orchestra.run.smithi059.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-16T19:54:30.175 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-16T19:54:30.236 INFO:teuthology.orchestra.run.smithi059.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-16T19:54:30.236 INFO:teuthology.orchestra.run.smithi059.stdout:This user must also own the server process. 2024-03-16T19:54:30.236 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:The database cluster will be initialized with locale "en_US". 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:The default text search configuration will be set to "english". 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:Data page checksums are disabled. 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:creating subdirectories ... ok 2024-03-16T19:54:30.237 INFO:teuthology.orchestra.run.smithi059.stdout:selecting dynamic shared memory implementation ... posix 2024-03-16T19:54:30.249 INFO:teuthology.orchestra.run.smithi059.stdout:selecting default max_connections ... 100 2024-03-16T19:54:30.272 INFO:teuthology.orchestra.run.smithi059.stdout:selecting default shared_buffers ... 128MB 2024-03-16T19:54:30.280 INFO:teuthology.orchestra.run.smithi059.stdout:selecting default time zone ... Etc/UTC 2024-03-16T19:54:30.282 INFO:teuthology.orchestra.run.smithi059.stdout:creating configuration files ... ok 2024-03-16T19:54:30.399 INFO:teuthology.orchestra.run.smithi059.stdout:running bootstrap script ... ok 2024-03-16T19:54:31.062 INFO:teuthology.orchestra.run.smithi059.stdout:performing post-bootstrap initialization ... ok 2024-03-16T19:54:31.650 INFO:teuthology.orchestra.run.smithi059.stdout:syncing data to disk ... ok 2024-03-16T19:54:32.033 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-16T19:54:34.772 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:54:34.898 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-16T19:54:35.024 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up postgresql (14+238) ... 2024-03-16T19:54:35.150 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-16T19:54:35.276 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-16T19:54:35.534 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-16T19:54:35.534 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-16T19:54:36.045 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-16T19:54:36.355 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-16T19:54:36.490 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-16T19:54:37.324 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-16T19:54:37.440 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-16T19:54:37.922 INFO:teuthology.orchestra.run.smithi059.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-16T19:54:37.927 DEBUG:teuthology.parallel:result is None 2024-03-16T19:54:37.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:54:38.961 DEBUG:teuthology.orchestra.run.smithi032:> dpkg-query -W -f '${Version}' ceph 2024-03-16T19:54:38.982 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:54:38.982 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:54:38.983 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681-gaba3c1d2-1jammy is installed. 2024-03-16T19:54:38.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:54:40.381 DEBUG:teuthology.orchestra.run.smithi059:> dpkg-query -W -f '${Version}' ceph 2024-03-16T19:54:40.402 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:54:40.402 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681-gaba3c1d2-1jammy 2024-03-16T19:54:40.402 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681-gaba3c1d2-1jammy is installed. 2024-03-16T19:54:40.404 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T19:54:40.404 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:40.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T19:54:40.419 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:40.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T19:54:40.456 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T19:54:40.456 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:40.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T19:54:40.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T19:54:40.532 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:40.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T19:54:40.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T19:54:40.599 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T19:54:40.599 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:40.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T19:54:40.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T19:54:40.668 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:40.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T19:54:40.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T19:54:40.738 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T19:54:40.739 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:40.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T19:54:40.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T19:54:40.808 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:40.809 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T19:54:40.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T19:54:40.875 INFO:teuthology.run_tasks:Running task ceph... 2024-03-16T19:54:40.962 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-16T19:54:40.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-03-16T19:54:40.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-03-16T19:54:40.979 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-16T19:54:40.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-16T19:54:41.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-16T19:54:41.035 INFO:tasks.ceph:Creating extra log directories... 2024-03-16T19:54:41.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-16T19:54:41.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-16T19:54:41.097 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-16T19:54:41.098 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-16T19:54:41.098 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-16T19:54:41.098 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:41.098 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-16T19:54:41.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-16T19:54:41.212 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:41.212 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-16T19:54:41.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-16T19:54:41.301 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-16T19:54:41.301 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-16T19:54:41.301 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118', 'branch': 'wip-lusov-qdb-squid', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7602118', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb-squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'valgrind': {'exit_on_first_error': False, 'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb-squid', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 6918, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'sleep_before_teardown': 0, 'subset': '111/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb-squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZw9i/4Vc9mop3CJjhqkAmxhPoQx9KrKTXVtPi6nmyvo7tsBHjJcg/SGsfC7PUNI7tjtIVIhxU0ZRID3L2wjFk=', 'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtaE+d2gd3u/ZI1NLSrq7vqbsLHzfwJBx7MfraW5/9rpHJZc1isETXaFsfab+PC6JveTnZ0pPQTlW8a5YAxKgY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'valgrind': {'exit_on_first_error': False, 'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-15_11:44:52', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-16T19:54:41.301 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-16T19:54:41.304 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-16T19:54:41.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-03-16T19:54:41.349 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-03-16T19:54:41.400 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:41.400 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T19:54:41.408 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T19:54:41.409 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-03-16T19:54:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T19:54:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:41.457 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-03-16T19:54:41.457 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:41.458 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-16 19:43:17.582818950 +0000 2024-03-16T19:54:41.458 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-16 19:43:17.342821093 +0000 2024-03-16T19:54:41.458 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-16 19:43:17.342821093 +0000 2024-03-16T19:54:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-03-16T19:54:41.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T19:54:41.511 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-16T19:54:41.511 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-16T19:54:41.512 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000319101 s, 1.6 MB/s 2024-03-16T19:54:41.513 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T19:54:41.563 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-16 19:43:18.070814591 +0000 2024-03-16T19:54:41.609 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-16 19:43:17.794817056 +0000 2024-03-16T19:54:41.610 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-16 19:43:17.794817056 +0000 2024-03-16T19:54:41.610 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-03-16T19:54:41.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T19:54:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-16T19:54:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-16T19:54:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.00029119 s, 1.8 MB/s 2024-03-16T19:54:41.665 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T19:54:41.714 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-03-16T19:54:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T19:54:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:41.761 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-03-16T19:54:41.761 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:41.761 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-16 19:43:18.610809767 +0000 2024-03-16T19:54:41.762 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-16 19:43:18.314812412 +0000 2024-03-16T19:54:41.762 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-16 19:43:18.314812412 +0000 2024-03-16T19:54:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-03-16T19:54:41.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T19:54:41.815 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-16T19:54:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-16T19:54:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000305821 s, 1.7 MB/s 2024-03-16T19:54:41.817 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T19:54:41.866 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-16 19:43:19.090805480 +0000 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-16 19:43:18.834807766 +0000 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-16 19:43:18.834807766 +0000 2024-03-16T19:54:41.917 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-03-16T19:54:41.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T19:54:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-16T19:54:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-16T19:54:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000249276 s, 2.1 MB/s 2024-03-16T19:54:41.975 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T19:54:42.022 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:42.022 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:42.022 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T19:54:42.028 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T19:54:42.028 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-03-16T19:54:42.075 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 870 Links: 1 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-16 19:43:17.355070949 +0000 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-16 19:43:17.351070985 +0000 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-16 19:43:17.351070985 +0000 2024-03-16T19:54:42.076 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-03-16T19:54:42.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T19:54:42.131 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-16T19:54:42.131 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-16T19:54:42.131 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000424912 s, 1.2 MB/s 2024-03-16T19:54:42.132 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T19:54:42.181 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-03-16T19:54:42.227 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T19:54:42.227 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-16 19:43:18.131064017 +0000 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-16 19:43:17.887066197 +0000 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-16 19:43:17.887066197 +0000 2024-03-16T19:54:42.228 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-03-16T19:54:42.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T19:54:42.282 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-16T19:54:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-16T19:54:42.283 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00041112 s, 1.2 MB/s 2024-03-16T19:54:42.284 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T19:54:42.333 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-16 19:43:18.659059302 +0000 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-16 19:43:18.383061767 +0000 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-16 19:43:18.383061767 +0000 2024-03-16T19:54:42.382 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-03-16T19:54:42.383 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T19:54:42.434 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-16T19:54:42.434 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-16T19:54:42.435 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000412797 s, 1.2 MB/s 2024-03-16T19:54:42.436 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T19:54:42.486 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-03-16T19:54:42.535 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-16 19:43:18.879057337 +0000 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-16 19:43:18.875057371 +0000 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-16 19:43:18.875057371 +0000 2024-03-16T19:54:42.536 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-03-16T19:54:42.536 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T19:54:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-16T19:54:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-16T19:54:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000378474 s, 1.4 MB/s 2024-03-16T19:54:42.592 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T19:54:42.641 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:42.641 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-16T19:54:42.641 INFO:tasks.ceph:Generating config... 2024-03-16T19:54:42.642 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-16T19:54:42.642 INFO:tasks.ceph:[client] debug client = 20 2024-03-16T19:54:42.643 INFO:tasks.ceph:[client] debug ms = 1 2024-03-16T19:54:42.643 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-16T19:54:42.643 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-16T19:54:42.643 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-16T19:54:42.643 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-16T19:54:42.643 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-16T19:54:42.643 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-16T19:54:42.644 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-16T19:54:42.644 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-16T19:54:42.644 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-16T19:54:42.644 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-16T19:54:42.644 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-16T19:54:42.645 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-16T19:54:42.645 INFO:tasks.ceph:Setting up mon.a... 2024-03-16T19:54:42.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-16T19:54:42.680 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-03-16T19:54:42.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-16T19:54:42.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:54:42.828 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.b', '172.21.15.59'), ('mon.c', '[v2:172.21.15.59:3301,v1:172.21.15.59:6790]')] 2024-03-16T19:54:42.829 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32,172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-16T19:54:42.829 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:42.830 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-16T19:54:42.878 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --add b 172.21.15.59 --addv c '[v2:172.21.15.59:3301,v1:172.21.15.59:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:42.963 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:42.963 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid fb85817b-8df0-4e27-a0bc-dc277b22543a 2024-03-16T19:54:42.963 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-03-16T19:54:42.963 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-03-16T19:54:42.963 INFO:teuthology.orchestra.run.smithi032.stdout:fsid fb85817b-8df0-4e27-a0bc-dc277b22543a 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-03-16T19:54:42.992872+0000 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-03-16T19:54:42.992872+0000 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.b 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2024-03-16T19:54:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-16T19:54:42.965 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-16T19:54:43.012 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fb85817b-8df0-4e27-a0bc-dc277b22543a... 2024-03-16T19:54:43.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-16T19:54:43.057 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-16T19:54:43.086 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-16T19:54:43.086 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-16T19:54:43.086 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-16T19:54:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-16T19:54:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790]" 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: mds beacon grace = 60 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: mds heartbeat grace = 60 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: osd heartbeat grace = 60 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = fb85817b-8df0-4e27-a0bc-dc277b22543a 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-16T19:54:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.092 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-16T19:54:43.093 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 20 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 20 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 10 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-16T19:54:43.094 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: osd fast shutdown = False 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-16T19:54:43.095 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-16T19:54:43.096 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-16T19:54:43.097 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-16T19:54:43.098 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd crush smoke test = False 2024-03-16T19:54:43.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-16T19:54:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-16T19:54:43.101 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f] 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-16T19:54:43.102 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-16T19:54:43.103 INFO:teuthology.orchestra.run.smithi032.stdout: mds valgrind exit = True 2024-03-16T19:54:43.103 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-16T19:54:43.103 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-16T19:54:43.103 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-16T19:54:43.103 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-16T19:54:43.104 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-16T19:54:43.104 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-16T19:54:43.105 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-16T19:54:43.106 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-16T19:54:43.107 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.32,172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790]" 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: mds beacon grace = 60 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: mds heartbeat grace = 60 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: osd heartbeat grace = 60 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = fb85817b-8df0-4e27-a0bc-dc277b22543a 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-16T19:54:43.108 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-16T19:54:43.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-03-16T19:54:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: osd fast shutdown = False 2024-03-16T19:54:43.111 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.112 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-16T19:54:43.113 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.114 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd crush smoke test = False 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-16T19:54:43.115 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.e] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.f] 2024-03-16T19:54:43.116 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: mds valgrind exit = True 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-16T19:54:43.117 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-16T19:54:43.117 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-16T19:54:43.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-16T19:54:43.198 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-16T19:54:43.199 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:43.199 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-16T19:54:43.205 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:43.205 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-16T19:54:43.256 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:54:43.257 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:43.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-16T19:54:43.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:54:43.322 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:54:43.322 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:43.372 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:54:43.373 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:43.373 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-16T19:54:43.373 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:54:43.396 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:54:43.396 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:43.444 INFO:tasks.ceph:Setting up mon nodes... 2024-03-16T19:54:43.444 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-16T19:54:43.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-16T19:54:43.490 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-16T19:54:43.492 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-16T19:54:43.538 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-16T19:54:43.553 INFO:tasks.ceph:Setting up mds nodes... 2024-03-16T19:54:43.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-16T19:54:43.594 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-16T19:54:43.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-16T19:54:43.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-16T19:54:43.735 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-16T19:54:43.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-16T19:54:43.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-16T19:54:43.878 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-16T19:54:43.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-16T19:54:43.937 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-16T19:54:43.982 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-16T19:54:43.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-16T19:54:44.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-16T19:54:44.128 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-16T19:54:44.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-16T19:54:44.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-16T19:54:44.271 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-16T19:54:44.274 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-16T19:54:44.327 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-16T19:54:44.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T19:54:44.365 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-16T19:54:44.377 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-16T19:54:44.377 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-16T19:54:44.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-16T19:54:44.435 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:44.436 INFO:tasks.ceph:role: osd.0 2024-03-16T19:54:44.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:54:44.436 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-16T19:54:44.494 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:44.494 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:44.494 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:44.495 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:44.518 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-16T19:54:44.520 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-16T19:54:44.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-16T19:54:44.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-16T19:54:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:44.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-16T19:54:44.690 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:44.690 INFO:tasks.ceph:role: osd.1 2024-03-16T19:54:44.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:54:44.690 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:44.748 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:44.771 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-16T19:54:44.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-16T19:54:44.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-16T19:54:44.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-16T19:54:44.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:44.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-16T19:54:44.948 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:44.948 INFO:tasks.ceph:role: osd.2 2024-03-16T19:54:44.948 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:54:44.948 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-16T19:54:45.004 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:45.004 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:45.028 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-16T19:54:45.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-16T19:54:45.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-16T19:54:45.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-16T19:54:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:45.142 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:45.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-16T19:54:45.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:45.196 INFO:tasks.ceph:role: osd.3 2024-03-16T19:54:45.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:54:45.196 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:45.277 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-16T19:54:45.278 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-16T19:54:45.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-16T19:54:45.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-16T19:54:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:45.391 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:45.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:45.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-16T19:54:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-16T19:54:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:54:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:54:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:54:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:45.529+0000 7f83a07827c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-16T19:54:48.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-16T19:54:48.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-16T19:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-16T19:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:48.601+0000 7f6dec4d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-16T19:54:51.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-16T19:54:51.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-16T19:54:51.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-16T19:54:51.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:54:51.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:54:51.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:54:51.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:51.693+0000 7f1d4c0cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-16T19:54:54.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-16T19:54:54.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:54:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.813+0000 7f791c5ff7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-16T19:54:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.817+0000 7f791c5ff7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-16T19:54:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.817+0000 7f791c5ff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:54:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.817+0000 7f791c5ff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:54:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.817+0000 7f791c5ff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:54:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:54:54.817+0000 7f791c5ff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-16T19:54:57.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-16T19:54:57.801 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-16T19:54:57.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:57.816 INFO:tasks.ceph:role: osd.4 2024-03-16T19:54:57.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:54:57.817 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:57.879 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:57.880 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:57.880 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:58.325 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-16T19:54:58.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-16T19:54:58.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-16T19:54:58.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-16T19:54:58.440 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:58.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-16T19:54:58.495 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:58.495 INFO:tasks.ceph:role: osd.5 2024-03-16T19:54:58.495 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:54:58.495 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-16T19:54:58.560 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:58.560 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:58.560 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:58.561 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:59.020 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-16T19:54:59.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-16T19:54:59.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-16T19:54:59.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-16T19:54:59.170 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:59.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-16T19:54:59.228 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:59.228 INFO:tasks.ceph:role: osd.6 2024-03-16T19:54:59.228 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:54:59.228 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-16T19:54:59.291 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:54:59.292 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:54:59.776 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-16T19:54:59.779 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-16T19:54:59.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-16T19:54:59.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-16T19:54:59.933 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:54:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:54:59.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-16T19:54:59.987 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:54:59.987 INFO:tasks.ceph:role: osd.7 2024-03-16T19:54:59.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T19:54:59.988 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:55:00.055 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:55:00.499 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-16T19:55:00.501 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-16T19:55:00.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-16T19:55:00.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-16T19:55:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:55:00.600 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-03-16T19:55:00.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:00.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.735+0000 7ff8498b37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-16T19:55:00.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.739+0000 7ff8498b37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-16T19:55:00.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.739+0000 7ff8498b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:55:00.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.739+0000 7ff8498b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:55:00.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.739+0000 7ff8498b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:55:00.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:00.739+0000 7ff8498b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-16T19:55:03.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-16T19:55:03.731 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:03.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-16T19:55:03.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-16T19:55:03.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:55:03.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:55:03.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:55:03.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:03.839+0000 7febfa7a97c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-16T19:55:06.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-16T19:55:06.855 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.959+0000 7fe7e5a407c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-16T19:55:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.963+0000 7fe7e5a407c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-16T19:55:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.963+0000 7fe7e5a407c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:55:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.963+0000 7fe7e5a407c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:55:06.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.963+0000 7fe7e5a407c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:55:06.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:06.963+0000 7fe7e5a407c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-16T19:55:09.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-16T19:55:09.984 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:10.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.087+0000 7f5585b2a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-16T19:55:10.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.087+0000 7f5585b2a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-16T19:55:10.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.087+0000 7f5585b2a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:55:10.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.087+0000 7f5585b2a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:55:10.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.087+0000 7f5585b2a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:55:10.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:10.091+0000 7f5585b2a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-16T19:55:11.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:55:11.306 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:55:13.042 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-16T19:55:13.056 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-16T19:55:13.057 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-16T19:55:13.071 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-16T19:55:13.128 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-16T19:55:13.184 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-16T19:55:13.240 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-16T19:55:13.295 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-16T19:55:13.352 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-16T19:55:13.407 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-16T19:55:13.463 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-16T19:55:13.477 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-16T19:55:13.531 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-16T19:55:13.586 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-16T19:55:13.642 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.642 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-16T19:55:13.702 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.703 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-16T19:55:13.758 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.759 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-16T19:55:13.814 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:13.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-16T19:55:13.870 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:13.870 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-16T19:55:13.876 INFO:tasks.ceph:Adding keys to all mons... 2024-03-16T19:55:13.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-16T19:55:13.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-16T19:55:13.930 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAD+fVl5+XzHhAAUX2LHdccHtmkbEVIhUlQeg== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAD+fVloRI5JRAAhf7rARzQzLedIPhV6kWCIg== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAD+fVlDvOVLRAA+61XwoDhJ0gjQIVZD3uWzg== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAD+fVlSPUjNhAA3YbBIEZwGS9/TGiOn43EXg== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAF+fVl+lukHxAAG+rBsKfOXJm+fon6hK2mKg== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAI+fVlxzvfIxAAxGG8HG4nAOMCCd6khH0axA== 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-03-16T19:55:13.931 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAL+fVl9xGKKRAA8h4ln1UkHodWUvhPXCZXnQ== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAO+fVliL68MBAAuXug+rviG2fRRlGeyup+RQ== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAD+fVlQIm+IRAAMumfL6eG8emOx0pJBbn+Tw== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAE+fVlmjKeABAAAhtPkUcfEtdWtbfVvgSNGg== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAE+fVlfgtLCRAAKEq0dA2t0AEfBdnfa1JIdg== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAE+fVlwMfVERAApN5yaBSPuyanQy44sE2Aag== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAU+fVlYP4wLBAA1lZcFRj0smG0AIXwzee0yQ== 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-03-16T19:55:13.932 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAX+fVlNEJTMhAAD2Rduad6OgmOuqpgIPFSZw== 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAa+fVlPpChORAAKz/JEfF6B8Ata7bRpYJq9w== 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAe+fVlbJqaBRAA3Oene/uRVN0KnsbevCskTQ== 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAE+fVlPY2KFxAA0rdqoH+spULif7wthagqpQ== 2024-03-16T19:55:13.933 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAD+fVl5+XzHhAAUX2LHdccHtmkbEVIhUlQeg== 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAD+fVloRI5JRAAhf7rARzQzLedIPhV6kWCIg== 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAD+fVlDvOVLRAA+61XwoDhJ0gjQIVZD3uWzg== 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.e] 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAD+fVlSPUjNhAA3YbBIEZwGS9/TGiOn43EXg== 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAF+fVl+lukHxAAG+rBsKfOXJm+fon6hK2mKg== 2024-03-16T19:55:13.934 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAI+fVlxzvfIxAAxGG8HG4nAOMCCd6khH0axA== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAL+fVl9xGKKRAA8h4ln1UkHodWUvhPXCZXnQ== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAO+fVliL68MBAAuXug+rviG2fRRlGeyup+RQ== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAD+fVlQIm+IRAAMumfL6eG8emOx0pJBbn+Tw== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAE+fVlmjKeABAAAhtPkUcfEtdWtbfVvgSNGg== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAE+fVlfgtLCRAAKEq0dA2t0AEfBdnfa1JIdg== 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.f] 2024-03-16T19:55:13.935 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAE+fVlwMfVERAApN5yaBSPuyanQy44sE2Aag== 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAU+fVlYP4wLBAA1lZcFRj0smG0AIXwzee0yQ== 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAX+fVlNEJTMhAAD2Rduad6OgmOuqpgIPFSZw== 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAa+fVlPpChORAAKz/JEfF6B8Ata7bRpYJq9w== 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAe+fVlbJqaBRAA3Oene/uRVN0KnsbevCskTQ== 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-16T19:55:13.936 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAE+fVlPY2KFxAA0rdqoH+spULif7wthagqpQ== 2024-03-16T19:55:13.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:55:13.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:55:14.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.031 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.293 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.373 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:55:14.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:55:14.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.854 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:55:14.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:14.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:15.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:15.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:15.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:15.157 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:55:15.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-16T19:55:15.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-16T19:55:15.285 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-16T19:55:15.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-16T19:55:15.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:55:15.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-16T19:55:15.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-16T19:55:15.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:55:15.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-16T19:55:15.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-16T19:55:15.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:55:15.709 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-16T19:55:15.719 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:15.722 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:55:15.766 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-16T19:55:15.766 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-16T19:55:15.766 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-16T19:55:15.767 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-16T19:55:15.769 INFO:tasks.ceph.mon.a:Started 2024-03-16T19:55:15.769 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-16T19:55:15.769 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-16T19:55:15.769 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-16T19:55:15.812 INFO:tasks.ceph.mon.b:Started 2024-03-16T19:55:15.812 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-16T19:55:15.812 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-16T19:55:15.812 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-16T19:55:15.814 INFO:tasks.ceph.mon.c:Started 2024-03-16T19:55:15.814 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-16T19:55:15.814 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-16T19:55:15.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-16T19:55:15.817 INFO:tasks.ceph.mgr.y:Started 2024-03-16T19:55:15.817 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-16T19:55:15.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-16T19:55:15.819 INFO:tasks.ceph.mgr.x:Started 2024-03-16T19:55:15.819 INFO:tasks.ceph:Setting crush tunables to default 2024-03-16T19:55:15.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-03-16T19:55:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 Processor -- start 2024-03-16T19:55:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 -- start start 2024-03-16T19:55:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040c3870 con 0x7f6c040cc5d0 2024-03-16T19:55:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6c04158420 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c09de1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cca80 con 0x7f6c04158800 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 msgr2=0x7f6c0415ee90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c02ffd640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6c040cc5d0 legacy=0x7f6c040c3160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 msgr2=0x7f6c04161810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:15.965+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.169+0000 7f6c02ffd640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6c040cc5d0 legacy=0x7f6c040c3160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.169+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 msgr2=0x7f6c0415ee90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.169+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.169+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 msgr2=0x7f6c04161810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.169+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.573+0000 7f6c02ffd640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6c040cc5d0 legacy=0x7f6c040c3160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.573+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 msgr2=0x7f6c0415ee90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.573+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.573+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 msgr2=0x7f6c04161810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:16.573+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:17.377+0000 7f6c02ffd640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6c040cc5d0 legacy=0x7f6c040c3160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:17.377+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 msgr2=0x7f6c0415ee90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:17.377+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:17.377+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 msgr2=0x7f6c04161810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:17.377+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 msgr2=0x7f6c04161810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c04158800 0x7f6c04161810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 msgr2=0x7f6c0415ee90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c04158420 0x7f6c0415ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6c040cc5d0 legacy=0x7f6c040c3160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040ccd60 con 0x7f6be4006130 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c017fa640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6be40036a0 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c037fe640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:18.965+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be4006130 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:19.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.169+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be4006130 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:19.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.169+0000 7f6c037fe640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:19.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.169+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.573+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be4006130 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.573+0000 7f6c037fe640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:19.573+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:20.377+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be4006130 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:20.377+0000 7f6c037fe640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:20.377+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be40036a0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be4006130 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c03fff640 1 -- reap_dead start 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040c3870 con 0x7f6be400a340 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6be4009ec0 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c017fa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cca80 con 0x7f6be400a890 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be400a340 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 msgr2=0x7f6be400d050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:21.965+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.169+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.169+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.169+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be400a340 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.169+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 msgr2=0x7f6be400d050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.169+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.573+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.573+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be400a340 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.573+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.573+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 msgr2=0x7f6be400d050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:22.573+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:23.377+0000 7f6c037fe640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-03-16T19:55:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:23.377+0000 7f6c02ffd640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be400a340 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2024-03-16T19:55:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:23.377+0000 7f6c037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:23.377+0000 7f6c03fff640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 msgr2=0x7f6be400d050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2024-03-16T19:55:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:23.377+0000 7f6c03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:24.510 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:08.607 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.538 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:00:08.573 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.566 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:00:08.603 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.854 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:08.952 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.883 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:00:08.919 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.911 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:00:08.947 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 msgr2=0x7f6be400d050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6be400a890 0x7f6be400d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f6be400a340 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040ccd60 con 0x7f6be400d6b0 2024-03-16T19:55:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c017fa640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6be400e270 2024-03-16T19:55:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c037fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c02ffd640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:24.965+0000 7f6c02ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.169+0000 7f6c037fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.169+0000 7f6c02ffd640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.169+0000 7f6c02ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.573+0000 7f6c02ffd640 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2024-03-16T19:55:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.573+0000 7f6c037fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-16T19:55:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:25.573+0000 7f6c02ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:26.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.493+0000 7f6c037fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 0x7f6be400dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38200/0 (socket says 172.21.15.32:38200) 2024-03-16T19:55:26.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.493+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 learned_addr learned my addr 172.21.15.32:0/182144487 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:55:26.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.553+0000 7f6c02ffd640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.613+0000 7f6c02ffd640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-16T19:55:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.613+0000 7f6c02ffd640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.613+0000 7f6c02ffd640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.613+0000 7f6c02ffd640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.841+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.841+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:26.841+0000 7f6c037fe640 1 --1- 172.21.15.32:0/182144487 >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 0x7f6be400dac0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.965+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.965+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be400e270 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.965+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.965+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 reap_dead start 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.965+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.969+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040c3870 con 0x7f6be400d6b0 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6be4009ec0 2024-03-16T19:55:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040cca80 con 0x7f6be4013ed0 2024-03-16T19:55:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:27.977+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:28.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.041+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.133+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.133+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.133+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.133+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.337+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.341+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.341+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.341+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.341+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.349+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-16T19:55:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.349+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.349+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.349+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:28.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.553+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.557+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.557+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.557+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.557+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:28.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.745+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.761+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.761+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.761+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:28.761+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.041+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:29.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.045+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.045+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.045+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.045+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:29.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.565+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:29.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.569+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:29.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.569+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:29.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.569+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:29.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.569+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:29.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.849+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:29.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.849+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:29.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.849+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:29.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.849+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:29.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:29.849+0000 7f6c03fff640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 msgr2=0x7f6be4013810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4013ed0 0x7f6be4013810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 msgr2=0x7f6be4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.59:3300/0 conn(0x7f6be4009ec0 0x7f6be4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 >> v1:172.21.15.32:6789/0 conn(0x7f6be400d6b0 legacy=0x7f6be400dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040ccd60 con 0x7f6be4014920 2024-03-16T19:55:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:30.969+0000 7f6c017fa640 1 -- 172.21.15.32:0/182144487 --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cd040 con 0x7f6be4014380 2024-03-16T19:55:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.001+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.005+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.005+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.005+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.005+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:55:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.209+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.213+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.213+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.213+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.213+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:55:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.681+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.801+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:55:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.801+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 msgr2=0x7f6be400d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:55:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.801+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:55:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:31.801+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:55:32.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.441+0000 7f6c027fc640 1 -- 172.21.15.32:0/182144487 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788465813 0 0) 0x7f6c040ccd60 con 0x7f6be4014920 2024-03-16T19:55:32.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.441+0000 7f6c027fc640 1 -- 172.21.15.32:0/182144487 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0003530 con 0x7f6be4014920 2024-03-16T19:55:32.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.605+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> v2:172.21.15.32:3300/0 conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:32.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.625+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 >> v1:172.21.15.59:6789/0 conn(0x7f6be4014920 legacy=0x7f6be40085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:32.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.625+0000 7f6c037fe640 1 -- 172.21.15.32:0/182144487 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be4015fc0 con 0x7f6be4014380 2024-03-16T19:55:32.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.657+0000 7f6c037fe640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be4014380 0x7f6be400d010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4003be0 tx=0x7f6bf401e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b370edd6ae5db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c027fc640 1 -- 172.21.15.32:0/182144487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40ad440 con 0x7f6be4014380 2024-03-16T19:55:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c027fc640 1 -- 172.21.15.32:0/182144487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf40ad5a0 con 0x7f6be4014380 2024-03-16T19:55:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c027fc640 1 -- 172.21.15.32:0/182144487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40ad930 con 0x7f6be4014380 2024-03-16T19:55:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 -- 172.21.15.32:0/182144487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be4014380 msgr2=0x7f6be400d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be4014380 0x7f6be400d010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4003be0 tx=0x7f6bf401e640 comp rx=0 tx=0).stop 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c03fff640 1 -- 172.21.15.32:0/182144487 reap_dead start 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 -- 172.21.15.32:0/182144487 shutdown_connections 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 --2- 172.21.15.32:0/182144487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be4014380 0x7f6be400d010 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 -- 172.21.15.32:0/182144487 >> 172.21.15.32:0/182144487 conn(0x7f6c04066ea0 msgr2=0x7f6c040721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 -- 172.21.15.32:0/182144487 shutdown_connections 2024-03-16T19:55:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.745+0000 7f6c09de1640 1 -- 172.21.15.32:0/182144487 wait complete. 2024-03-16T19:55:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 Processor -- start 2024-03-16T19:55:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 -- start start 2024-03-16T19:55:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c041f4170 0x7f6c041f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c040cb490 0x7f6c040cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041f3c70 con 0x7f6c041f4170 2024-03-16T19:55:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041f7fc0 con 0x7f6c040cb490 2024-03-16T19:55:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.749+0000 7f6c09de1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c04152680 con 0x7f6be4014380 2024-03-16T19:55:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.761+0000 7f6c02ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c041f4170 0x7f6c041f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.761+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.761+0000 7f6c037fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:36886/0 (socket says 172.21.15.32:36886) 2024-03-16T19:55:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.761+0000 7f6c037fe640 1 -- 172.21.15.32:0/2212077218 learned_addr learned my addr 172.21.15.32:0/2212077218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:55:32.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.769+0000 7f6c037fe640 1 -- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c040cb490 msgr2=0x7f6c040cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:32.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.769+0000 7f6c037fe640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c040cb490 0x7f6c040cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.769+0000 7f6c037fe640 1 -- 172.21.15.32:0/2212077218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c041f4170 msgr2=0x7f6c041f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.769+0000 7f6c037fe640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c041f4170 0x7f6c041f4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.769+0000 7f6c037fe640 1 -- 172.21.15.32:0/2212077218 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf400f040 con 0x7f6be4014380 2024-03-16T19:55:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.781+0000 7f6c037fe640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4008240 tx=0x7f6bf40083d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:32.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.853+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4004330 con 0x7f6be4014380 2024-03-16T19:55:32.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.853+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf40ba070 con 0x7f6be4014380 2024-03-16T19:55:32.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.853+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40ab410 con 0x7f6be4014380 2024-03-16T19:55:32.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.853+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c04152960 con 0x7f6be4014380 2024-03-16T19:55:32.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.853+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c04152f20 con 0x7f6be4014380 2024-03-16T19:55:32.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.857+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c041530b0 con 0x7f6be4014380 2024-03-16T19:55:32.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:32.893+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6bf40ab900 con 0x7f6be4014380 2024-03-16T19:55:33.074 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:33.101+0000 7f68ffce9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T19:55:33.144 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:33.167+0000 7f6f2944f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T19:55:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.177+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6bf40c99f0 con 0x7f6be4014380 2024-03-16T19:55:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.181+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6bf40bc070 con 0x7f6be4014380 2024-03-16T19:55:33.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.273+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6c04068000 con 0x7f6be4014380 2024-03-16T19:55:33.573 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:33.601+0000 7f68ffce9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T19:55:33.649 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:33.671+0000 7f6f2944f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T19:55:33.785 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:33.813+0000 7f68ffce9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T19:55:33.863 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:33.886+0000 7f6f2944f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T19:55:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6bfbfff640 1 -- 172.21.15.32:0/2212077218 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6bf40c9450 con 0x7f6be4014380 2024-03-16T19:55:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 msgr2=0x7f6c0414cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4008240 tx=0x7f6bf40083d0 comp rx=0 tx=0).stop 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 shutdown_connections 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c040cb490 0x7f6c040cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c041f4170 0x7f6c041f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 --2- 172.21.15.32:0/2212077218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6be4014380 0x7f6c0414cbd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 >> 172.21.15.32:0/2212077218 conn(0x7f6c04066ea0 msgr2=0x7f6c040721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 shutdown_connections 2024-03-16T19:55:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:33.949+0000 7f6c09de1640 1 -- 172.21.15.32:0/2212077218 wait complete. 2024-03-16T19:55:33.942 INFO:tasks.ceph:check_enable_crimson: False 2024-03-16T19:55:33.942 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-16T19:55:33.942 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:33.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-16T19:55:33.953 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:33.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-16T19:55:34.006 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:34.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-16T19:55:34.058 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:55:34.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-16T19:55:34.115 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:34.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-16T19:55:34.128 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:34.128 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-16T19:55:34.183 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:34.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-16T19:55:34.241 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-16T19:55:34.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-16T19:55:34.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 2b55a720-9297-4595-89bb-0e9a1f3da471 0 2024-03-16T19:55:34.370 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.397+0000 7f68ffce9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T19:55:34.427 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.453+0000 7f68ffce9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T19:55:34.451 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.474+0000 7f6f2944f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T19:55:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 Processor -- start 2024-03-16T19:55:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 -- start start 2024-03-16T19:55:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2dc151cc0 0x7fb2dc0c9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc067320 con 0x7fb2dc0c7040 2024-03-16T19:55:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc06bbc0 con 0x7fb2dc151cc0 2024-03-16T19:55:34.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.486+0000 7fb2e0afb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc157c20 con 0x7fb2dc158780 2024-03-16T19:55:34.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.490+0000 7fb2dad9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:34.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.490+0000 7fb2dad9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55142/0 (socket says 172.21.15.59:55142) 2024-03-16T19:55:34.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.490+0000 7fb2dad9d640 1 -- 172.21.15.59:0/3735938324 learned_addr learned my addr 172.21.15.59:0/3735938324 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:34.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.494+0000 7fb2dad9d640 1 -- 172.21.15.59:0/3735938324 >> v2:172.21.15.32:3300/0 conn(0x7fb2dc151cc0 msgr2=0x7fb2dc0c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:34.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.494+0000 7fb2dad9d640 1 --2- 172.21.15.59:0/3735938324 >> v2:172.21.15.32:3300/0 conn(0x7fb2dc151cc0 0x7fb2dc0c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:34.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.494+0000 7fb2dad9d640 1 -- 172.21.15.59:0/3735938324 >> v1:172.21.15.32:6789/0 conn(0x7fb2dc0c7040 legacy=0x7fb2dc158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:34.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.494+0000 7fb2dad9d640 1 -- 172.21.15.59:0/3735938324 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0704d0 con 0x7fb2dc158780 2024-03-16T19:55:34.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.502+0000 7fb2dad9d640 1 --2- 172.21.15.59:0/3735938324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d401c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3656fc06381ce86b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:34.482 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.509+0000 7f68ffce9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T19:55:34.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2d959a640 1 -- 172.21.15.59:0/3735938324 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40a57b0 con 0x7fb2dc158780 2024-03-16T19:55:34.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2d959a640 1 -- 172.21.15.59:0/3735938324 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d40a5910 con 0x7fb2dc158780 2024-03-16T19:55:34.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2d959a640 1 -- 172.21.15.59:0/3735938324 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40a5ca0 con 0x7fb2dc158780 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3735938324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 msgr2=0x7fb2dc0c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3735938324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d401c930 comp rx=0 tx=0).stop 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3735938324 shutdown_connections 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3735938324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc158780 0x7fb2dc0c74b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3735938324 >> v2:172.21.15.32:3300/0 conn(0x7fb2dc151cc0 0x7fb2dc0c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3735938324 >> 172.21.15.59:0/3735938324 conn(0x7fb2dc066ea0 msgr2=0x7fb2dc0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3735938324 shutdown_connections 2024-03-16T19:55:34.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3735938324 wait complete. 2024-03-16T19:55:34.513 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.534+0000 7f6f2944f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T19:55:34.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 Processor -- start 2024-03-16T19:55:34.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.534+0000 7fb2e0afb640 1 -- start start 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2dc151cc0 0x7fb2dc14cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2dc158780 0x7fb2dc14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc14cbd0 con 0x7fb2dc151cc0 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc14c5e0 con 0x7fb2dc158780 2024-03-16T19:55:34.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2e0afb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc14c050 con 0x7fb2dc0c7040 2024-03-16T19:55:34.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:34.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2dad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2dc158780 0x7fb2dc14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:34.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55156/0 (socket says 172.21.15.59:55156) 2024-03-16T19:55:34.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.538+0000 7fb2da59c640 1 -- 172.21.15.59:0/3688421678 learned_addr learned my addr 172.21.15.59:0/3688421678 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:34.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.542+0000 7fb2da59c640 1 -- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2dc158780 msgr2=0x7fb2dc14d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:34.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.542+0000 7fb2da59c640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2dc158780 0x7fb2dc14d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:34.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.542+0000 7fb2da59c640 1 -- 172.21.15.59:0/3688421678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2dc151cc0 msgr2=0x7fb2dc14cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:34.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.542+0000 7fb2da59c640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2dc151cc0 0x7fb2dc14cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:34.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.542+0000 7fb2da59c640 1 -- 172.21.15.59:0/3688421678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4009b30 con 0x7fb2dc0c7040 2024-03-16T19:55:34.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2da59c640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00b580 tx=0x7fb2cc00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:34.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc010870 con 0x7fb2dc0c7040 2024-03-16T19:55:34.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc0109f0 con 0x7fb2dc0c7040 2024-03-16T19:55:34.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc14c2f0 con 0x7fb2dc0c7040 2024-03-16T19:55:34.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc010da0 con 0x7fb2dc0c7040 2024-03-16T19:55:34.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.546+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc13ecf0 con 0x7fb2dc0c7040 2024-03-16T19:55:34.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.550+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc14ce40 con 0x7fb2dc0c7040 2024-03-16T19:55:34.563 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.593+0000 7f68ffce9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T19:55:34.583 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.606+0000 7f6f2944f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T19:55:34.618 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.645+0000 7f68ffce9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T19:55:34.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.686+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb2cc023480 con 0x7fb2dc0c7040 2024-03-16T19:55:34.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.686+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb2cc019b20 con 0x7fb2dc0c7040 2024-03-16T19:55:34.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.686+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb2cc01e060 con 0x7fb2dc0c7040 2024-03-16T19:55:34.667 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.690+0000 7f6f2944f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T19:55:34.698 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.725+0000 7f68ffce9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T19:55:34.724 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.746+0000 7f6f2944f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T19:55:34.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:34.770+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2b55a720-9297-4595-89bb-0e9a1f3da471", "id": 0} v 0) v1 -- 0x7fb2dc068000 con 0x7fb2dc0c7040 2024-03-16T19:55:34.754 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.781+0000 7f68ffce9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T19:55:34.812 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.834+0000 7f6f2944f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T19:55:34.872 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:34.894+0000 7f6f2944f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T19:55:34.910 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:34.937+0000 7f68ffce9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T19:55:34.975 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:35.005+0000 7f68ffce9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T19:55:35.040 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:35.062+0000 7f6f2944f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T19:55:35.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2caffd640 1 -- 172.21.15.59:0/3688421678 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b55a720-9297-4595-89bb-0e9a1f3da471", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2cc014070 con 0x7fb2dc0c7040 2024-03-16T19:55:35.087 INFO:teuthology.orchestra.run.smithi059.stdout:0 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 msgr2=0x7fb2dc143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00b580 tx=0x7fb2cc00ba40 comp rx=0 tx=0).stop 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 shutdown_connections 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2dc158780 0x7fb2dc14d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2dc151cc0 0x7fb2dc14cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 --2- 172.21.15.59:0/3688421678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2dc0c7040 0x7fb2dc143100 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 >> 172.21.15.59:0/3688421678 conn(0x7fb2dc066ea0 msgr2=0x7fb2dc158070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 shutdown_connections 2024-03-16T19:55:35.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.110+0000 7fb2e0afb640 1 -- 172.21.15.59:0/3688421678 wait complete. 2024-03-16T19:55:35.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 30704413-befa-4817-92b0-d0f96b5d354d 1 2024-03-16T19:55:35.116 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:35.138+0000 7f6f2944f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T19:55:35.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.246+0000 7ff318a2e640 1 Processor -- start 2024-03-16T19:55:35.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.246+0000 7ff318a2e640 1 -- start start 2024-03-16T19:55:35.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff318a2e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff3140c8d00 0x7ff3141636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff318a2e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140cdc60 con 0x7ff3140705b0 2024-03-16T19:55:35.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff318a2e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140cdf40 con 0x7ff314151fc0 2024-03-16T19:55:35.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff318a2e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3140ce500 con 0x7ff3140c8d00 2024-03-16T19:55:35.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff31259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff314151fc0 0x7ff31415b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55060/0 (socket says 172.21.15.59:55060) 2024-03-16T19:55:35.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff31259c640 1 -- 172.21.15.59:0/3213935275 learned_addr learned my addr 172.21.15.59:0/3213935275 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-16T19:55:35.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.250+0000 7ff312d9d640 1 --2- 172.21.15.59:0/3213935275 >> v2:172.21.15.32:3300/0 conn(0x7ff3140c8d00 0x7ff3141636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.254+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169062804 0 0) 0x7ff3140cdf40 con 0x7ff314151fc0 2024-03-16T19:55:35.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.254+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003530 con 0x7ff314151fc0 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2439191633 0 0) 0x7ff2f8003530 con 0x7ff314151fc0 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 >> v2:172.21.15.32:3300/0 conn(0x7ff3140c8d00 msgr2=0x7ff3141636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 --2- 172.21.15.59:0/3213935275 >> v2:172.21.15.32:3300/0 conn(0x7ff3140c8d00 0x7ff3141636c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 >> v1:172.21.15.32:6789/0 conn(0x7ff3140705b0 legacy=0x7ff3140c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140ce220 con 0x7ff314151fc0 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7ff304001680 con 0x7ff314151fc0 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 >> v1:172.21.15.59:6789/0 conn(0x7ff314151fc0 legacy=0x7ff31415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2f80039d0 0x7ff2f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140ce500 con 0x7ff2f80039d0 2024-03-16T19:55:35.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800c040 con 0x7ff2f8005190 2024-03-16T19:55:35.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800c200 con 0x7ff2f80047d0 2024-03-16T19:55:35.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 -- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2f8005190 msgr2=0x7ff2f800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 -- 172.21.15.59:0/3213935275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2f80039d0 msgr2=0x7ff2f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2f80039d0 0x7ff2f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff311d9b640 1 -- 172.21.15.59:0/3213935275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8003530 con 0x7ff2f80047d0 2024-03-16T19:55:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.258+0000 7ff312d9d640 1 -- 172.21.15.59:0/3213935275 reap_dead start 2024-03-16T19:55:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.262+0000 7ff311d9b640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff300001650 tx=0x7ff30001c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862a900d10069fec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.266+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300005270 con 0x7ff2f80047d0 2024-03-16T19:55:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.266+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3000a7070 con 0x7ff2f80047d0 2024-03-16T19:55:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.266+0000 7ff31159a640 1 -- 172.21.15.59:0/3213935275 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300003860 con 0x7ff2f80047d0 2024-03-16T19:55:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.266+0000 7ff318a2e640 1 -- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 msgr2=0x7ff2f8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.266+0000 7ff318a2e640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff300001650 tx=0x7ff30001c2e0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- 172.21.15.59:0/3213935275 shutdown_connections 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 --2- 172.21.15.59:0/3213935275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- 172.21.15.59:0/3213935275 >> 172.21.15.59:0/3213935275 conn(0x7ff314066ea0 msgr2=0x7ff3140672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- 172.21.15.59:0/3213935275 shutdown_connections 2024-03-16T19:55:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- 172.21.15.59:0/3213935275 wait complete. 2024-03-16T19:55:35.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 Processor -- start 2024-03-16T19:55:35.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- start start 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3140cd200 0x7ff31414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3140c7450 0x7ff31414d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140c7a00 con 0x7ff3140cd200 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31414fdf0 con 0x7ff3140c7450 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff318a2e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31414a5d0 con 0x7ff2f80047d0 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff31259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff312d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3140c7450 0x7ff31414d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff31259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55176/0 (socket says 172.21.15.59:55176) 2024-03-16T19:55:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.270+0000 7ff31259c640 1 -- 172.21.15.59:0/911186969 learned_addr learned my addr 172.21.15.59:0/911186969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff311d9b640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3140cd200 0x7ff31414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff31259c640 1 -- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3140c7450 msgr2=0x7ff31414d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff31259c640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3140c7450 0x7ff31414d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff31259c640 1 -- 172.21.15.59:0/911186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3140cd200 msgr2=0x7ff31414cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff31259c640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3140cd200 0x7ff31414cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.274+0000 7ff31259c640 1 -- 172.21.15.59:0/911186969 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3141ee8a0 con 0x7ff2f80047d0 2024-03-16T19:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.278+0000 7ff31259c640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff30400f790 tx=0x7ff30400fb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:35.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304007af0 con 0x7ff2f80047d0 2024-03-16T19:55:35.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304007c50 con 0x7ff2f80047d0 2024-03-16T19:55:35.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30400a0d0 con 0x7ff2f80047d0 2024-03-16T19:55:35.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3140c6700 con 0x7ff2f80047d0 2024-03-16T19:55:35.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3141ee530 con 0x7ff2f80047d0 2024-03-16T19:55:35.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.282+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3141ee320 con 0x7ff2f80047d0 2024-03-16T19:55:35.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.286+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff30400a250 con 0x7ff2f80047d0 2024-03-16T19:55:35.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.286+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7ff3040207f0 con 0x7ff2f80047d0 2024-03-16T19:55:35.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.406+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff304025060 con 0x7ff2f80047d0 2024-03-16T19:55:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.526+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "30704413-befa-4817-92b0-d0f96b5d354d", "id": 1} v 0) v1 -- 0x7ff314068000 con 0x7ff2f80047d0 2024-03-16T19:55:35.658 INFO:tasks.ceph.mgr.y.smithi032.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-16T19:55:35.659 INFO:tasks.ceph.mgr.y.smithi032.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-16T19:55:35.659 INFO:tasks.ceph.mgr.y.smithi032.stderr: from numpy import show_config as show_numpy_config 2024-03-16T19:55:35.701 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:35.729+0000 7f68ffce9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T19:55:35.766 INFO:tasks.ceph.mgr.x.smithi059.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-16T19:55:35.766 INFO:tasks.ceph.mgr.x.smithi059.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-16T19:55:35.766 INFO:tasks.ceph.mgr.x.smithi059.stderr: from numpy import show_config as show_numpy_config 2024-03-16T19:55:35.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff30affd640 1 -- 172.21.15.59:0/911186969 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "30704413-befa-4817-92b0-d0f96b5d354d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff30401b070 con 0x7ff2f80047d0 2024-03-16T19:55:35.788 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-16T19:55:35.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 msgr2=0x7ff31415b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff30400f790 tx=0x7ff30400fb90 comp rx=0 tx=0).stop 2024-03-16T19:55:35.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 shutdown_connections 2024-03-16T19:55:35.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3140c7450 0x7ff31414d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3140cd200 0x7ff31414cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 --2- 172.21.15.59:0/911186969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff2f80047d0 0x7ff31415b030 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 >> 172.21.15.59:0/911186969 conn(0x7ff314066ea0 msgr2=0x7ff3140b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:35.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 shutdown_connections 2024-03-16T19:55:35.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.810+0000 7ff318a2e640 1 -- 172.21.15.59:0/911186969 wait complete. 2024-03-16T19:55:35.809 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 18500384-71d8-40a4-a508-b47407b9d67c 2 2024-03-16T19:55:35.825 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:35.846+0000 7f6f2944f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T19:55:35.853 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:35.881+0000 7f68ffce9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:55:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.970+0000 7fe8b4ab5640 1 Processor -- start 2024-03-16T19:55:35.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.970+0000 7fe8b4ab5640 1 -- start start 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8b4ab5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b006ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8b4ab5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8b00ce0f0 0x7fe8b0070510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8b4ab5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b006cd90 con 0x7fe8b0159fa0 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8b4ab5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b015a450 con 0x7fe8b00ce0f0 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8b4ab5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b015a730 con 0x7fe8b00cdd10 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8ae59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b006ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8a7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe8b0159fa0 0x7fe8b015bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55068/0 (socket says 172.21.15.59:55068) 2024-03-16T19:55:35.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8a7fff640 1 -- 172.21.15.59:0/3245598921 learned_addr learned my addr 172.21.15.59:0/3245598921 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-16T19:55:35.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.974+0000 7fe8aed9d640 1 --2- 172.21.15.59:0/3245598921 >> v2:172.21.15.32:3300/0 conn(0x7fe8b00ce0f0 0x7fe8b0070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.978+0000 7fe8ae59c640 1 -- 172.21.15.59:0/3245598921 >> v2:172.21.15.32:3300/0 conn(0x7fe8b00ce0f0 msgr2=0x7fe8b0070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:35.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.978+0000 7fe8ae59c640 1 --2- 172.21.15.59:0/3245598921 >> v2:172.21.15.32:3300/0 conn(0x7fe8b00ce0f0 0x7fe8b0070510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.978+0000 7fe8ae59c640 1 -- 172.21.15.59:0/3245598921 >> v1:172.21.15.59:6789/0 conn(0x7fe8b0159fa0 legacy=0x7fe8b015bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.978+0000 7fe8ae59c640 1 -- 172.21.15.59:0/3245598921 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b015aa10 con 0x7fe8b00cdd10 2024-03-16T19:55:35.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.978+0000 7fe8add9b640 1 -- 172.21.15.59:0/3245598921 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212929842 0 0) 0x7fe8b006cd90 con 0x7fe8b0159fa0 2024-03-16T19:55:35.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.982+0000 7fe8ae59c640 1 --2- 172.21.15.59:0/3245598921 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b006ffd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097f0 tx=0x7fe89801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2496c18e38a045e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8add9b640 1 -- 172.21.15.59:0/3245598921 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980a8990 con 0x7fe8b00cdd10 2024-03-16T19:55:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8add9b640 1 -- 172.21.15.59:0/3245598921 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8980a8af0 con 0x7fe8b00cdd10 2024-03-16T19:55:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8add9b640 1 -- 172.21.15.59:0/3245598921 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980a8e80 con 0x7fe8b00cdd10 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/3245598921 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 msgr2=0x7fe8b006ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/3245598921 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b006ffd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097f0 tx=0x7fe89801ec90 comp rx=0 tx=0).stop 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/3245598921 shutdown_connections 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/3245598921 >> v2:172.21.15.32:3300/0 conn(0x7fe8b00ce0f0 0x7fe8b0070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/3245598921 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b006ffd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/3245598921 >> 172.21.15.59:0/3245598921 conn(0x7fe8b0066ea0 msgr2=0x7fe8b00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/3245598921 shutdown_connections 2024-03-16T19:55:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/3245598921 wait complete. 2024-03-16T19:55:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 Processor -- start 2024-03-16T19:55:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.986+0000 7fe8b4ab5640 1 -- start start 2024-03-16T19:55:35.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b00ce0f0 0x7fe8b01f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8b0159fa0 0x7fe8b00af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b0154330 con 0x7fe8b0159fa0 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b01f3170 con 0x7fe8b00ce0f0 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8b4ab5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c8270 con 0x7fe8b00cdd10 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8ae59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b00ce0f0 0x7fe8b01f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8ae59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55188/0 (socket says 172.21.15.59:55188) 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8ae59c640 1 -- 172.21.15.59:0/801472292 learned_addr learned my addr 172.21.15.59:0/801472292 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:35.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.990+0000 7fe8aed9d640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8b0159fa0 0x7fe8b00af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 -- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b00ce0f0 msgr2=0x7fe8b01f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b00ce0f0 0x7fe8b01f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 -- 172.21.15.59:0/801472292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8b0159fa0 msgr2=0x7fe8b00af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8b0159fa0 0x7fe8b00af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 -- 172.21.15.59:0/801472292 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898009420 con 0x7fe8b00cdd10 2024-03-16T19:55:35.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.994+0000 7fe8ae59c640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe89801c8c0 tx=0x7fe89801ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:35.974 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:35.998+0000 7f6f2944f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:55:35.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.998+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980b9630 con 0x7fe8b00cdd10 2024-03-16T19:55:35.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.998+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8980b97b0 con 0x7fe8b00cdd10 2024-03-16T19:55:35.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.998+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b01f0600 con 0x7fe8b00cdd10 2024-03-16T19:55:35.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.998+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980b9b60 con 0x7fe8b00cdd10 2024-03-16T19:55:35.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:35.998+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b0153d70 con 0x7fe8b00cdd10 2024-03-16T19:55:35.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.002+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b0068000 con 0x7fe8b00cdd10 2024-03-16T19:55:35.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.002+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe8980b7040 con 0x7fe8b00cdd10 2024-03-16T19:55:35.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.002+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe8980b9cc0 con 0x7fe8b00cdd10 2024-03-16T19:55:36.025 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:36.053+0000 7f68ffce9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T19:55:36.092 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:36.121+0000 7f68ffce9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T19:55:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.118+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe8980b5070 con 0x7fe8b00cdd10 2024-03-16T19:55:36.142 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:36.162+0000 7f6f2944f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T19:55:36.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.194+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "18500384-71d8-40a4-a508-b47407b9d67c", "id": 2} v 0) v1 -- 0x7fe8b00af960 con 0x7fe8b00cdd10 2024-03-16T19:55:36.210 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:36.230+0000 7f6f2944f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T19:55:36.245 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:36.273+0000 7f68ffce9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:55:36.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8a6ffd640 1 -- 172.21.15.59:0/801472292 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18500384-71d8-40a4-a508-b47407b9d67c", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe8980d20b0 con 0x7fe8b00cdd10 2024-03-16T19:55:36.323 INFO:teuthology.orchestra.run.smithi059.stdout:2 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 msgr2=0x7fe8b01f00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe89801c8c0 tx=0x7fe89801ba30 comp rx=0 tx=0).stop 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 shutdown_connections 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8b0159fa0 0x7fe8b00af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b00ce0f0 0x7fe8b01f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 --2- 172.21.15.59:0/801472292 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8b00cdd10 0x7fe8b01f00c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 >> 172.21.15.59:0/801472292 conn(0x7fe8b0066ea0 msgr2=0x7fe8b00b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 shutdown_connections 2024-03-16T19:55:36.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.346+0000 7fe8b4ab5640 1 -- 172.21.15.59:0/801472292 wait complete. 2024-03-16T19:55:36.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new b04758b0-8a06-46e4-8061-232f252de437 3 2024-03-16T19:55:36.357 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:36.378+0000 7f6f2944f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:55:36.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 Processor -- start 2024-03-16T19:55:36.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 -- start start 2024-03-16T19:55:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23cc072bb0 0x7f23cc071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc156500 con 0x7f23cc150560 2024-03-16T19:55:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc0c1560 con 0x7f23cc072bb0 2024-03-16T19:55:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d3f33640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc150940 con 0x7f23cc0c2e10 2024-03-16T19:55:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d24d0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.482+0000 7f23d14ce640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23cc072bb0 0x7f23cc071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:36.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d24d0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55200/0 (socket says 172.21.15.59:55200) 2024-03-16T19:55:36.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d24d0640 1 -- 172.21.15.59:0/4042922790 learned_addr learned my addr 172.21.15.59:0/4042922790 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d14ce640 1 -- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 msgr2=0x7f23cc072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d14ce640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d14ce640 1 -- 172.21.15.59:0/4042922790 >> v1:172.21.15.32:6789/0 conn(0x7f23cc150560 legacy=0x7f23cc156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d14ce640 1 -- 172.21.15.59:0/4042922790 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0c69a0 con 0x7f23cc072bb0 2024-03-16T19:55:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.486+0000 7f23d24d0640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:55:36.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.490+0000 7f23d14ce640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc072bb0 0x7f23cc071e50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0097c0 tx=0x7f23bc01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed968fcdb91f7fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:36.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.494+0000 7f23d0ccd640 1 -- 172.21.15.59:0/4042922790 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0ab600 con 0x7f23cc072bb0 2024-03-16T19:55:36.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.494+0000 7f23d0ccd640 1 -- 172.21.15.59:0/4042922790 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc0a7050 con 0x7f23cc072bb0 2024-03-16T19:55:36.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.494+0000 7f23d0ccd640 1 -- 172.21.15.59:0/4042922790 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0ab980 con 0x7f23cc072bb0 2024-03-16T19:55:36.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc072bb0 msgr2=0x7f23cc071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:36.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc072bb0 0x7f23cc071e50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0097c0 tx=0x7f23bc01c460 comp rx=0 tx=0).stop 2024-03-16T19:55:36.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- 172.21.15.59:0/4042922790 shutdown_connections 2024-03-16T19:55:36.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc0c2e10 0x7f23cc072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- 172.21.15.59:0/4042922790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc072bb0 0x7f23cc071e50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- 172.21.15.59:0/4042922790 >> 172.21.15.59:0/4042922790 conn(0x7f23cc06f3d0 msgr2=0x7f23cc06f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:36.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- 172.21.15.59:0/4042922790 shutdown_connections 2024-03-16T19:55:36.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- 172.21.15.59:0/4042922790 wait complete. 2024-03-16T19:55:36.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 Processor -- start 2024-03-16T19:55:36.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- start start 2024-03-16T19:55:36.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:36.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc0c2e10 0x7f23cc1f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:36.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc150560 0x7f23cc14b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:36.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc14ec10 con 0x7f23cc0c2e10 2024-03-16T19:55:36.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc141a40 con 0x7f23cc150560 2024-03-16T19:55:36.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d3f33640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc1f3b90 con 0x7f23cc072bb0 2024-03-16T19:55:36.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d1ccf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:36.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d24d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc150560 0x7f23cc14b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:36.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d1ccf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55204/0 (socket says 172.21.15.59:55204) 2024-03-16T19:55:36.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.498+0000 7f23d1ccf640 1 -- 172.21.15.59:0/1796116706 learned_addr learned my addr 172.21.15.59:0/1796116706 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:36.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.502+0000 7f23d14ce640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc0c2e10 0x7f23cc1f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:36.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 -- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc150560 msgr2=0x7f23cc14b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:36.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc150560 0x7f23cc14b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 -- 172.21.15.59:0/1796116706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc0c2e10 msgr2=0x7f23cc1f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:36.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc0c2e10 0x7f23cc1f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 -- 172.21.15.59:0/1796116706 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4009b30 con 0x7f23cc072bb0 2024-03-16T19:55:36.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.506+0000 7f23d1ccf640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23c800b410 tx=0x7f23c800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.510+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8011830 con 0x7f23cc072bb0 2024-03-16T19:55:36.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.510+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8011990 con 0x7f23cc072bb0 2024-03-16T19:55:36.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.510+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8011d40 con 0x7f23cc072bb0 2024-03-16T19:55:36.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.510+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc009420 con 0x7f23cc072bb0 2024-03-16T19:55:36.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.510+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc14f430 con 0x7f23cc072bb0 2024-03-16T19:55:36.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.514+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc0c6bf0 con 0x7f23cc072bb0 2024-03-16T19:55:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.514+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f23c80247d0 con 0x7f23cc072bb0 2024-03-16T19:55:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.514+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f23c801abb0 con 0x7f23cc072bb0 2024-03-16T19:55:36.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.630+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f23c8039dd0 con 0x7f23cc072bb0 2024-03-16T19:55:36.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.706+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b04758b0-8a06-46e4-8061-232f252de437", "id": 3} v 0) v1 -- 0x7f23cc1418a0 con 0x7f23cc072bb0 2024-03-16T19:55:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.890+0000 7f23c27fc640 1 -- 172.21.15.59:0/1796116706 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b04758b0-8a06-46e4-8061-232f252de437", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f23cc1418a0 con 0x7f23cc072bb0 2024-03-16T19:55:36.870 INFO:teuthology.orchestra.run.smithi059.stdout:3 2024-03-16T19:55:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 msgr2=0x7f23cc14e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23c800b410 tx=0x7f23c800b8d0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 shutdown_connections 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23cc150560 0x7f23cc14b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc0c2e10 0x7f23cc1f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 --2- 172.21.15.59:0/1796116706 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23cc072bb0 0x7f23cc14e6d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 >> 172.21.15.59:0/1796116706 conn(0x7f23cc06f3d0 msgr2=0x7f23cc154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 shutdown_connections 2024-03-16T19:55:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:36.894+0000 7f23d3f33640 1 -- 172.21.15.59:0/1796116706 wait complete. 2024-03-16T19:55:36.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 3358c3b9-7e70-42e6-b285-3865a95d8b19 4 2024-03-16T19:55:36.920 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:36.949+0000 7f68ffce9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T19:55:36.999 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:37.025+0000 7f68ffce9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T19:55:37.022 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.042+0000 7f6f2944f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T19:55:37.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 Processor -- start 2024-03-16T19:55:37.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 -- start start 2024-03-16T19:55:37.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35c00c8d00 0x7f35c00c8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00cdc60 con 0x7f35c0151fc0 2024-03-16T19:55:37.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c00ce500 con 0x7f35c00c8d00 2024-03-16T19:55:37.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c8278640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00b2d40 con 0x7f35c00705b0 2024-03-16T19:55:37.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c5813640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c6815640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35c00c8d00 0x7f35c00c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c5813640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55214/0 (socket says 172.21.15.59:55214) 2024-03-16T19:55:37.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.074+0000 7f35c5813640 1 -- 172.21.15.59:0/3208444073 learned_addr learned my addr 172.21.15.59:0/3208444073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:37.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.078+0000 7f35c5813640 1 -- 172.21.15.59:0/3208444073 >> v2:172.21.15.59:3300/0 conn(0x7f35c00c8d00 msgr2=0x7f35c00c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:37.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.078+0000 7f35c5813640 1 --2- 172.21.15.59:0/3208444073 >> v2:172.21.15.59:3300/0 conn(0x7f35c00c8d00 0x7f35c00c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.078+0000 7f35c5813640 1 -- 172.21.15.59:0/3208444073 >> v1:172.21.15.59:6789/0 conn(0x7f35c0151fc0 legacy=0x7f35c015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.078+0000 7f35c5813640 1 -- 172.21.15.59:0/3208444073 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c00cdf40 con 0x7f35c00705b0 2024-03-16T19:55:37.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.082+0000 7f35c5813640 1 --2- 172.21.15.59:0/3208444073 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009870 tx=0x7f35b401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed54908460695961 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:37.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c5012640 1 -- 172.21.15.59:0/3208444073 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40a8a50 con 0x7f35c00705b0 2024-03-16T19:55:37.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c5012640 1 -- 172.21.15.59:0/3208444073 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b40a8bb0 con 0x7f35c00705b0 2024-03-16T19:55:37.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c5012640 1 -- 172.21.15.59:0/3208444073 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40b0440 con 0x7f35c00705b0 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 -- 172.21.15.59:0/3208444073 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 msgr2=0x7f35c00c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 --2- 172.21.15.59:0/3208444073 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009870 tx=0x7f35b401ed50 comp rx=0 tx=0).stop 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 -- 172.21.15.59:0/3208444073 shutdown_connections 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 --2- 172.21.15.59:0/3208444073 >> v2:172.21.15.59:3300/0 conn(0x7f35c00c8d00 0x7f35c00c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 --2- 172.21.15.59:0/3208444073 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c00705b0 0x7f35c00c8100 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 -- 172.21.15.59:0/3208444073 >> 172.21.15.59:0/3208444073 conn(0x7f35c0066ea0 msgr2=0x7f35c00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 -- 172.21.15.59:0/3208444073 shutdown_connections 2024-03-16T19:55:37.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.086+0000 7f35c8278640 1 -- 172.21.15.59:0/3208444073 wait complete. 2024-03-16T19:55:37.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 Processor -- start 2024-03-16T19:55:37.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 -- start start 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c00705b0 0x7f35c006c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c00c8d00 0x7f35c0153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c0157f20 con 0x7f35c00705b0 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c01582b0 con 0x7f35c00c8d00 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c8278640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c0156240 con 0x7f35c0151fc0 2024-03-16T19:55:37.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c6815640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c5813640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c00c8d00 0x7f35c0153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c6815640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55218/0 (socket says 172.21.15.59:55218) 2024-03-16T19:55:37.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.090+0000 7f35c6815640 1 -- 172.21.15.59:0/27271041 learned_addr learned my addr 172.21.15.59:0/27271041 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:37.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6014640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c00705b0 0x7f35c006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6815640 1 -- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c00c8d00 msgr2=0x7f35c0153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6815640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c00c8d00 0x7f35c0153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6815640 1 -- 172.21.15.59:0/27271041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c00705b0 msgr2=0x7f35c006c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6815640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c00705b0 0x7f35c006c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.094+0000 7f35c6815640 1 -- 172.21.15.59:0/27271041 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40094d0 con 0x7f35c0151fc0 2024-03-16T19:55:37.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.098+0000 7f35c6815640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f35b000efc0 tx=0x7f35b000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0009ca0 con 0x7f35c0151fc0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000f040 con 0x7f35c0151fc0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b00146c0 con 0x7f35c0151fc0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c0158710 con 0x7f35c0151fc0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c0156490 con 0x7f35c0151fc0 2024-03-16T19:55:37.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.102+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c0068000 con 0x7f35c0151fc0 2024-03-16T19:55:37.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.106+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f35b0014bb0 con 0x7f35c0151fc0 2024-03-16T19:55:37.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.106+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f35b00138f0 con 0x7f35c0151fc0 2024-03-16T19:55:37.110 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.130+0000 7f6f2944f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T19:55:37.188 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:37.217+0000 7f68ffce9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T19:55:37.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.222+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f35b003bdd0 con 0x7f35c0151fc0 2024-03-16T19:55:37.265 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:37.293+0000 7f68ffce9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T19:55:37.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.294+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3358c3b9-7e70-42e6-b285-3865a95d8b19", "id": 4} v 0) v1 -- 0x7f35c00c2de0 con 0x7f35c0151fc0 2024-03-16T19:55:37.300 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.322+0000 7f6f2944f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T19:55:37.379 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.402+0000 7f6f2944f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T19:55:37.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.530+0000 7f35beffd640 1 -- 172.21.15.59:0/27271041 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3358c3b9-7e70-42e6-b285-3865a95d8b19", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f35b002a030 con 0x7f35c0151fc0 2024-03-16T19:55:37.511 INFO:teuthology.orchestra.run.smithi059.stdout:4 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 msgr2=0x7f35c006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f35b000efc0 tx=0x7f35b000c490 comp rx=0 tx=0).stop 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 shutdown_connections 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c0151fc0 0x7f35c006d7e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c00c8d00 0x7f35c0153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 --2- 172.21.15.59:0/27271041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c00705b0 0x7f35c006c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 >> 172.21.15.59:0/27271041 conn(0x7f35c0066ea0 msgr2=0x7f35c0067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 shutdown_connections 2024-03-16T19:55:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.534+0000 7f35c8278640 1 -- 172.21.15.59:0/27271041 wait complete. 2024-03-16T19:55:37.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 28be6473-1ce4-448f-bfa9-437370b18219 5 2024-03-16T19:55:37.575 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:37.605+0000 7f68ffce9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T19:55:37.640 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:37.669+0000 7f68ffce9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T19:55:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 Processor -- start 2024-03-16T19:55:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 -- start start 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b70150f00 0x7f6b700c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b700c7040 0x7f6b700c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b700c3540 0x7f6b70156b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b70070850 con 0x7f6b700c7040 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b700c7e50 con 0x7f6b700c3540 2024-03-16T19:55:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b77ff5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701512e0 con 0x7f6b70150f00 2024-03-16T19:55:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b76592640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b700c3540 0x7f6b70156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b75d91640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b70150f00 0x7f6b700c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b76592640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b700c3540 0x7f6b70156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34906/0 (socket says 172.21.15.59:34906) 2024-03-16T19:55:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.674+0000 7f6b76592640 1 -- 172.21.15.59:0/3537280926 learned_addr learned my addr 172.21.15.59:0/3537280926 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.678+0000 7f6b76592640 1 -- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b70150f00 msgr2=0x7f6b700c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.678+0000 7f6b76592640 1 --2- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b70150f00 0x7f6b700c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.678+0000 7f6b76592640 1 -- 172.21.15.59:0/3537280926 >> v2:172.21.15.32:3300/0 conn(0x7f6b700c7040 msgr2=0x7f6b700c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.678+0000 7f6b76592640 1 --2- 172.21.15.59:0/3537280926 >> v2:172.21.15.32:3300/0 conn(0x7f6b700c7040 0x7f6b700c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.678+0000 7f6b76592640 1 -- 172.21.15.59:0/3537280926 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70070290 con 0x7f6b700c3540 2024-03-16T19:55:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.682+0000 7f6b76592640 1 --2- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b700c3540 0x7f6b70156b40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b7006d8d0 tx=0x7f6b6801c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d855587234f5f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:37.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b74d8f640 1 -- 172.21.15.59:0/3537280926 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b680a5780 con 0x7f6b700c3540 2024-03-16T19:55:37.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b74d8f640 1 -- 172.21.15.59:0/3537280926 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b680a58e0 con 0x7f6b700c3540 2024-03-16T19:55:37.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b74d8f640 1 -- 172.21.15.59:0/3537280926 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b680a5c70 con 0x7f6b700c3540 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b700c3540 msgr2=0x7f6b70156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b700c3540 0x7f6b70156b40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b7006d8d0 tx=0x7f6b6801c9f0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- 172.21.15.59:0/3537280926 shutdown_connections 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b700c3540 0x7f6b70156b40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/3537280926 >> v2:172.21.15.32:3300/0 conn(0x7f6b700c7040 0x7f6b700c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/3537280926 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b70150f00 0x7f6b700c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- 172.21.15.59:0/3537280926 >> 172.21.15.59:0/3537280926 conn(0x7f6b70066ea0 msgr2=0x7f6b700721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- 172.21.15.59:0/3537280926 shutdown_connections 2024-03-16T19:55:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- 172.21.15.59:0/3537280926 wait complete. 2024-03-16T19:55:37.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 Processor -- start 2024-03-16T19:55:37.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- start start 2024-03-16T19:55:37.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b700c3540 0x7f6b70157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b70150f00 0x7f6b7013ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70159110 con 0x7f6b700c3540 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70142740 con 0x7f6b70150f00 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b77ff5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701598e0 con 0x7f6b700c7040 2024-03-16T19:55:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.686+0000 7f6b75590640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:37.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.690+0000 7f6b75590640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55236/0 (socket says 172.21.15.59:55236) 2024-03-16T19:55:37.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.690+0000 7f6b75590640 1 -- 172.21.15.59:0/2734530186 learned_addr learned my addr 172.21.15.59:0/2734530186 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.694+0000 7f6b75590640 1 -- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b70150f00 msgr2=0x7f6b7013ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.694+0000 7f6b75590640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b70150f00 0x7f6b7013ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.694+0000 7f6b75590640 1 -- 172.21.15.59:0/2734530186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b700c3540 msgr2=0x7f6b70157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.694+0000 7f6b75590640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b700c3540 0x7f6b70157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.694+0000 7f6b75590640 1 -- 172.21.15.59:0/2734530186 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b68009b30 con 0x7f6b700c7040 2024-03-16T19:55:37.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.698+0000 7f6b75590640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000b580 tx=0x7f6b6000ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:37.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60010870 con 0x7f6b700c7040 2024-03-16T19:55:37.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b600109f0 con 0x7f6b700c7040 2024-03-16T19:55:37.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60010da0 con 0x7f6b700c7040 2024-03-16T19:55:37.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b70150730 con 0x7f6b700c7040 2024-03-16T19:55:37.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b70142c50 con 0x7f6b700c7040 2024-03-16T19:55:37.707 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6f2944f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T19:55:37.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.730+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b70068000 con 0x7f6b700c7040 2024-03-16T19:55:37.775 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-03-16T19:55:37.798+0000 7f6f2944f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T19:55:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.846+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6b60023480 con 0x7f6b700c7040 2024-03-16T19:55:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.846+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f6b6000f940 con 0x7f6b700c7040 2024-03-16T19:55:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.846+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6b6001e060 con 0x7f6b700c7040 2024-03-16T19:55:37.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:37.926+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "28be6473-1ce4-448f-bfa9-437370b18219", "id": 5} v 0) v1 -- 0x7f6b7014d490 con 0x7f6b700c7040 2024-03-16T19:55:38.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:38.550+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mgrmap(e 2) v1 ==== 43246+0+0 (secure 0 0 0) 0x7f6b6000fb50 con 0x7f6b700c7040 2024-03-16T19:55:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b667fc640 1 -- 172.21.15.59:0/2734530186 <== mon.2 v2:172.21.15.59:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28be6473-1ce4-448f-bfa9-437370b18219", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6b60014020 con 0x7f6b700c7040 2024-03-16T19:55:39.029 INFO:teuthology.orchestra.run.smithi059.stdout:5 2024-03-16T19:55:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 msgr2=0x7f6b70153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000b580 tx=0x7f6b6000ba40 comp rx=0 tx=0).stop 2024-03-16T19:55:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 shutdown_connections 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b70150f00 0x7f6b7013ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b700c7040 0x7f6b70153c80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 --2- 172.21.15.59:0/2734530186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b700c3540 0x7f6b70157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 >> 172.21.15.59:0/2734530186 conn(0x7f6b70066ea0 msgr2=0x7f6b7014df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.050+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 shutdown_connections 2024-03-16T19:55:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.054+0000 7f6b77ff5640 1 -- 172.21.15.59:0/2734530186 wait complete. 2024-03-16T19:55:39.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 22cff389-1c2e-4b39-914c-f06fcf60a0c6 6 2024-03-16T19:55:39.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.222+0000 7fc7a7ba2640 1 Processor -- start 2024-03-16T19:55:39.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.222+0000 7fc7a7ba2640 1 -- start start 2024-03-16T19:55:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.226+0000 7fc7a7ba2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7a00c7040 0x7fc7a00c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.226+0000 7fc7a7ba2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7a0158070 0x7fc7a01584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.226+0000 7fc7a7ba2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a0067320 con 0x7fc7a0151cc0 2024-03-16T19:55:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.226+0000 7fc7a7ba2640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a006bbc0 con 0x7fc7a0158070 2024-03-16T19:55:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.226+0000 7fc7a7ba2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a0157c20 con 0x7fc7a00c7040 2024-03-16T19:55:39.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.230+0000 7fc7a513d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7a00c7040 0x7fc7a00c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:39.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.230+0000 7fc7a593e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc7a0151cc0 0x7fc7a00c9820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55994/0 (socket says 172.21.15.59:55994) 2024-03-16T19:55:39.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.230+0000 7fc7a593e640 1 -- 172.21.15.59:0/1908774748 learned_addr learned my addr 172.21.15.59:0/1908774748 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-16T19:55:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.282+0000 7fc7a513d640 1 -- 172.21.15.59:0/1908774748 >> v2:172.21.15.32:3300/0 conn(0x7fc7a0158070 msgr2=0x7fc7a01584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.282+0000 7fc7a513d640 1 --2- 172.21.15.59:0/1908774748 >> v2:172.21.15.32:3300/0 conn(0x7fc7a0158070 0x7fc7a01584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.282+0000 7fc7a513d640 1 -- 172.21.15.59:0/1908774748 >> v1:172.21.15.59:6789/0 conn(0x7fc7a0151cc0 legacy=0x7fc7a00c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.282+0000 7fc7a513d640 1 -- 172.21.15.59:0/1908774748 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a00c9f30 con 0x7fc7a00c7040 2024-03-16T19:55:39.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.286+0000 7fc7a513d640 1 --2- 172.21.15.59:0/1908774748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a00c74b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00c8ff0 tx=0x7fc79001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70540cd8fae07d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.294+0000 7fc7a493c640 1 -- 172.21.15.59:0/1908774748 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900ab680 con 0x7fc7a00c7040 2024-03-16T19:55:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.294+0000 7fc7a493c640 1 -- 172.21.15.59:0/1908774748 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7900ab7e0 con 0x7fc7a00c7040 2024-03-16T19:55:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a493c640 1 -- 172.21.15.59:0/1908774748 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900abb70 con 0x7fc7a00c7040 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/1908774748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 msgr2=0x7fc7a00c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/1908774748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a00c74b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00c8ff0 tx=0x7fc79001c480 comp rx=0 tx=0).stop 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/1908774748 shutdown_connections 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/1908774748 >> v2:172.21.15.32:3300/0 conn(0x7fc7a0158070 0x7fc7a01584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/1908774748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a00c74b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/1908774748 >> 172.21.15.59:0/1908774748 conn(0x7fc7a0066ea0 msgr2=0x7fc7a00721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/1908774748 shutdown_connections 2024-03-16T19:55:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.298+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/1908774748 wait complete. 2024-03-16T19:55:39.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 Processor -- start 2024-03-16T19:55:39.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 -- start start 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 0x7fc7a013e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7a0158070 0x7fc7a014e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01f65a0 con 0x7fc7a0158070 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01f6b00 con 0x7fc7a00c7040 2024-03-16T19:55:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a7ba2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00c57a0 con 0x7fc7a0151cc0 2024-03-16T19:55:39.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a593e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:39.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a513d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 0x7fc7a013e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:39.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a593e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a014f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37500/0 (socket says 172.21.15.59:37500) 2024-03-16T19:55:39.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.302+0000 7fc7a593e640 1 -- 172.21.15.59:0/2562951714 learned_addr learned my addr 172.21.15.59:0/2562951714 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:39.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.306+0000 7fc7a513d640 1 -- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 msgr2=0x7fc7a014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:39.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.306+0000 7fc7a513d640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.306+0000 7fc7a513d640 1 -- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7a0158070 msgr2=0x7fc7a014e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:39.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.306+0000 7fc7a513d640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7a0158070 0x7fc7a014e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:39.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.306+0000 7fc7a513d640 1 -- 172.21.15.59:0/2562951714 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790009420 con 0x7fc7a0151cc0 2024-03-16T19:55:39.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.310+0000 7fc7a513d640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 0x7fc7a013e950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7900ac000 tx=0x7fc79001acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:39.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900aaba0 con 0x7fc7a0151cc0 2024-03-16T19:55:39.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7900aad20 con 0x7fc7a0151cc0 2024-03-16T19:55:39.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900bf650 con 0x7fc7a0151cc0 2024-03-16T19:55:39.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a00c5a40 con 0x7fc7a0151cc0 2024-03-16T19:55:39.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a0155aa0 con 0x7fc7a0151cc0 2024-03-16T19:55:39.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.314+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a0155c90 con 0x7fc7a0151cc0 2024-03-16T19:55:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.326+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 2) v1 ==== 43246+0+0 (secure 0 0 0) 0x7fc7900b5400 con 0x7fc7a0151cc0 2024-03-16T19:55:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.326+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc7900aae90 con 0x7fc7a0151cc0 2024-03-16T19:55:39.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.450+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc7900c83f0 con 0x7fc7a0151cc0 2024-03-16T19:55:39.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:39.558+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "22cff389-1c2e-4b39-914c-f06fcf60a0c6", "id": 6} v 0) v1 -- 0x7fc7a0068000 con 0x7fc7a0151cc0 2024-03-16T19:55:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.246+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mgrmap(e 3) v1 ==== 86247+0+0 (secure 0 0 0) 0x7fc7900b5d20 con 0x7fc7a0151cc0 2024-03-16T19:55:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.246+0000 7fc7967fc640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 0x7fc7840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.246+0000 7fc7a593e640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 0x7fc7840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:40.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.246+0000 7fc7a593e640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 0x7fc7840693e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc798001f60 tx=0x7fc798001ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:40.299 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:40.329+0000 7f68a7239640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-16T19:55:40.417 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T19:55:40.445+0000 7f68a7239640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-16T19:55:40.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.858+0000 7fc7967fc640 1 -- 172.21.15.59:0/2562951714 <== mon.2 v2:172.21.15.59:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22cff389-1c2e-4b39-914c-f06fcf60a0c6", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7900dac80 con 0x7fc7a0151cc0 2024-03-16T19:55:40.837 INFO:teuthology.orchestra.run.smithi059.stdout:6 2024-03-16T19:55:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 msgr2=0x7fc7840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 0x7fc7840693e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc798001f60 tx=0x7fc798001ad0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 msgr2=0x7fc7a013e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 0x7fc7a013e950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7900ac000 tx=0x7fc79001acc0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 shutdown_connections 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc784066ff0 0x7fc7840693e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7a0158070 0x7fc7a014e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7a0151cc0 0x7fc7a013e950 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 --2- 172.21.15.59:0/2562951714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00c7040 0x7fc7a014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 >> 172.21.15.59:0/2562951714 conn(0x7fc7a0066ea0 msgr2=0x7fc7a0157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 shutdown_connections 2024-03-16T19:55:40.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:40.862+0000 7fc7a7ba2640 1 -- 172.21.15.59:0/2562951714 wait complete. 2024-03-16T19:55:40.865 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 4c2d369f-f6e2-4644-b752-e10b0d42117b 7 2024-03-16T19:55:40.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.014+0000 7fee86c69640 1 Processor -- start 2024-03-16T19:55:40.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.014+0000 7fee86c69640 1 -- start start 2024-03-16T19:55:40.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee86c69640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee80070f60 0x7fee800ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:40.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee86c69640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800cd5a0 con 0x7fee801523c0 2024-03-16T19:55:40.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee86c69640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee80158110 con 0x7fee800cadb0 2024-03-16T19:55:40.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee86c69640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee80157f00 con 0x7fee80070f60 2024-03-16T19:55:40.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee77fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fee800cadb0 0x7fee80151cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56006/0 (socket says 172.21.15.59:56006) 2024-03-16T19:55:40.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.018+0000 7fee77fff640 1 -- 172.21.15.59:0/2381836415 learned_addr learned my addr 172.21.15.59:0/2381836415 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-16T19:55:41.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.106+0000 7fee84a05640 1 --2- 172.21.15.59:0/2381836415 >> v2:172.21.15.32:3300/0 conn(0x7fee80070f60 0x7fee800ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.106+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640580087 0 0) 0x7fee800cd5a0 con 0x7fee801523c0 2024-03-16T19:55:41.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.106+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee68003530 con 0x7fee801523c0 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116816985 0 0) 0x7fee68003530 con 0x7fee801523c0 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 >> v2:172.21.15.32:3300/0 conn(0x7fee80070f60 msgr2=0x7fee800ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 --2- 172.21.15.59:0/2381836415 >> v2:172.21.15.32:3300/0 conn(0x7fee80070f60 0x7fee800ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 >> v1:172.21.15.59:6789/0 conn(0x7fee800cadb0 legacy=0x7fee80151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee801583f0 con 0x7fee801523c0 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fee78001a00 con 0x7fee801523c0 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 >> v1:172.21.15.32:6789/0 conn(0x7fee801523c0 legacy=0x7fee800cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 0x7fee68003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680047d0 0x7fee68004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee68005390 0x7fee6800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80157f00 con 0x7fee68005390 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6800c200 con 0x7fee680047d0 2024-03-16T19:55:41.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6800c3c0 con 0x7fee680039d0 2024-03-16T19:55:41.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.114+0000 7fee84a05640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 0x7fee68003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee77fff640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680047d0 0x7fee68004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee84a05640 1 -- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680047d0 msgr2=0x7fee68004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:41.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee84a05640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680047d0 0x7fee68004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee84a05640 1 -- 172.21.15.59:0/2381836415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee68005390 msgr2=0x7fee6800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:41.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee84a05640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee68005390 0x7fee6800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee84a05640 1 -- 172.21.15.59:0/2381836415 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee68003530 con 0x7fee680039d0 2024-03-16T19:55:41.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.118+0000 7fee85206640 1 -- 172.21.15.59:0/2381836415 reap_dead start 2024-03-16T19:55:41.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.126+0000 7fee84a05640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 0x7fee68003df0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00a990 tx=0x7fee7c01c5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=807b9545e11e05ec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.134+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0aa660 con 0x7fee680039d0 2024-03-16T19:55:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.134+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c0aa7c0 con 0x7fee680039d0 2024-03-16T19:55:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.134+0000 7fee777fe640 1 -- 172.21.15.59:0/2381836415 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0aabb0 con 0x7fee680039d0 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 msgr2=0x7fee68003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 0x7fee68003df0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00a990 tx=0x7fee7c01c5b0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- 172.21.15.59:0/2381836415 shutdown_connections 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 --2- 172.21.15.59:0/2381836415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee680039d0 0x7fee68003df0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- 172.21.15.59:0/2381836415 >> 172.21.15.59:0/2381836415 conn(0x7fee8006f5e0 msgr2=0x7fee8006e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- 172.21.15.59:0/2381836415 shutdown_connections 2024-03-16T19:55:41.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- 172.21.15.59:0/2381836415 wait complete. 2024-03-16T19:55:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 Processor -- start 2024-03-16T19:55:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.138+0000 7fee86c69640 1 -- start start 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee680039d0 0x7fee800c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee801f3550 0x7fee80153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80153a40 con 0x7fee680039d0 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee801effe0 con 0x7fee801535c0 2024-03-16T19:55:41.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee86c69640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee801f02a0 con 0x7fee801f3550 2024-03-16T19:55:41.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee85206640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee77fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee801f3550 0x7fee80153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee85206640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37516/0 (socket says 172.21.15.59:37516) 2024-03-16T19:55:41.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.142+0000 7fee85206640 1 -- 172.21.15.59:0/988442333 learned_addr learned my addr 172.21.15.59:0/988442333 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-16T19:55:41.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.146+0000 7fee85206640 1 -- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee801f3550 msgr2=0x7fee80153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:41.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.146+0000 7fee85206640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee801f3550 0x7fee80153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.146+0000 7fee85206640 1 -- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee680039d0 msgr2=0x7fee800c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:41.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.146+0000 7fee85206640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee680039d0 0x7fee800c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:41.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.146+0000 7fee85206640 1 -- 172.21.15.59:0/988442333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee801f0580 con 0x7fee801535c0 2024-03-16T19:55:41.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.150+0000 7fee85206640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fee800c9f50 tx=0x7fee7800ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7800a610 con 0x7fee801535c0 2024-03-16T19:55:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7800a790 con 0x7fee801535c0 2024-03-16T19:55:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee801f0860 con 0x7fee801535c0 2024-03-16T19:55:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7800ab40 con 0x7fee801535c0 2024-03-16T19:55:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee801f0e20 con 0x7fee801535c0 2024-03-16T19:55:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.154+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee80152ed0 con 0x7fee801535c0 2024-03-16T19:55:41.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:55:41.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:55:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.586+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 3) v1 ==== 86247+0+0 (secure 0 0 0) 0x7fee7800aca0 con 0x7fee801535c0 2024-03-16T19:55:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.586+0000 7fee757fa640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 0x7fee6c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:41.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.586+0000 7fee84a05640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 0x7fee6c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:41.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.586+0000 7fee84a05640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 0x7fee6c0691d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fee7c0ae000 tx=0x7fee7c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:41.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.618+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fee78055030 con 0x7fee801535c0 2024-03-16T19:55:41.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.618+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fee780512d0 con 0x7fee801535c0 2024-03-16T19:55:41.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.814+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 4) v1 ==== 86353+0+0 (secure 0 0 0) 0x7fee780516a0 con 0x7fee801535c0 2024-03-16T19:55:41.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:41.862+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4c2d369f-f6e2-4644-b752-e10b0d42117b", "id": 7} v 0) v1 -- 0x7fee801f0fb0 con 0x7fee801535c0 2024-03-16T19:55:42.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.522+0000 7fee757fa640 1 -- 172.21.15.59:0/988442333 <== mon.1 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4c2d369f-f6e2-4644-b752-e10b0d42117b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fee780170b0 con 0x7fee801535c0 2024-03-16T19:55:42.502 INFO:teuthology.orchestra.run.smithi059.stdout:7 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 msgr2=0x7fee6c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 0x7fee6c0691d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fee7c0ae000 tx=0x7fee7c004740 comp rx=0 tx=0).stop 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 msgr2=0x7fee8006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fee800c9f50 tx=0x7fee7800ccd0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 shutdown_connections 2024-03-16T19:55:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee6c066d10 0x7fee6c0691d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee801535c0 0x7fee8006d6a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fee801f3550 0x7fee80153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 --2- 172.21.15.59:0/988442333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee680039d0 0x7fee800c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 >> 172.21.15.59:0/988442333 conn(0x7fee8006f5e0 msgr2=0x7fee8006e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 shutdown_connections 2024-03-16T19:55:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-16T19:55:42.526+0000 7fee86c69640 1 -- 172.21.15.59:0/988442333 wait complete. 2024-03-16T19:55:42.528 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-16T19:55:42.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-16T19:55:42.531 INFO:tasks.ceph.osd.0:Started 2024-03-16T19:55:42.531 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-16T19:55:42.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-16T19:55:42.533 INFO:tasks.ceph.osd.1:Started 2024-03-16T19:55:42.533 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-16T19:55:42.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-16T19:55:42.536 INFO:tasks.ceph.osd.2:Started 2024-03-16T19:55:42.536 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-16T19:55:42.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-16T19:55:42.538 INFO:tasks.ceph.osd.3:Started 2024-03-16T19:55:42.539 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-16T19:55:42.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-16T19:55:42.541 INFO:tasks.ceph.osd.4:Started 2024-03-16T19:55:42.541 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-16T19:55:42.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-16T19:55:42.544 INFO:tasks.ceph.osd.5:Started 2024-03-16T19:55:42.544 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-16T19:55:42.544 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-16T19:55:42.546 INFO:tasks.ceph.osd.6:Started 2024-03-16T19:55:42.546 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-16T19:55:42.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-16T19:55:42.549 INFO:tasks.ceph.osd.7:Started 2024-03-16T19:55:42.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:55:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 Processor -- start 2024-03-16T19:55:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 -- start start 2024-03-16T19:55:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 0x7f7f4c158340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c157eb0 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0c4c80 con 0x7f7f4c0cdb80 2024-03-16T19:55:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f5177f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c06fa80 con 0x7f7f4c0c7830 2024-03-16T19:55:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f4a7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7f4c0cdf60 0x7f7f4c151b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36262/0 (socket says 172.21.15.32:36262) 2024-03-16T19:55:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f4a7fc640 1 -- 172.21.15.32:0/1043951114 learned_addr learned my addr 172.21.15.32:0/1043951114 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:55:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.713+0000 7f7f4b7fe640 1 --2- 172.21.15.32:0/1043951114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 0x7f7f4c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946649880 0 0) 0x7f7f4c157eb0 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f34003530 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/1043951114 >> v1:172.21.15.59:6789/0 conn(0x7f7f4c0cdb80 legacy=0x7f7f4c06f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/1043951114 >> v1:172.21.15.32:6789/0 conn(0x7f7f4c0cdf60 legacy=0x7f7f4c151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138817878 0 0) 0x7f7f4c0c4c80 con 0x7f7f4c0cdb80 2024-03-16T19:55:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.717+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/1043951114 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0c4f40 con 0x7f7f4c0c7830 2024-03-16T19:55:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.721+0000 7f7f4b7fe640 1 --2- 172.21.15.32:0/1043951114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 0x7f7f4c158340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009dc0 tx=0x7f7f4401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde80b5262e1f175 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.725+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440a5930 con 0x7f7f4c0c7830 2024-03-16T19:55:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.725+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f440a5a90 con 0x7f7f4c0c7830 2024-03-16T19:55:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.725+0000 7f7f49ffb640 1 -- 172.21.15.32:0/1043951114 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440a5e20 con 0x7f7f4c0c7830 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- 172.21.15.32:0/1043951114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 msgr2=0x7f7f4c158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 --2- 172.21.15.32:0/1043951114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 0x7f7f4c158340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009dc0 tx=0x7f7f4401c960 comp rx=0 tx=0).stop 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- 172.21.15.32:0/1043951114 shutdown_connections 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 --2- 172.21.15.32:0/1043951114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0c7830 0x7f7f4c158340 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- 172.21.15.32:0/1043951114 >> 172.21.15.32:0/1043951114 conn(0x7f7f4c066ea0 msgr2=0x7f7f4c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- 172.21.15.32:0/1043951114 shutdown_connections 2024-03-16T19:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- 172.21.15.32:0/1043951114 wait complete. 2024-03-16T19:55:42.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 Processor -- start 2024-03-16T19:55:42.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- start start 2024-03-16T19:55:42.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0c7830 0x7f7f4c1f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0cdb80 0x7f7f4c1457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1f6e40 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c149b90 con 0x7f7f4c0c7830 2024-03-16T19:55:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f5177f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1f6ba0 con 0x7f7f4c0cdb80 2024-03-16T19:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f4b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f4a7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0cdb80 0x7f7f4c1457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f4affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0c7830 0x7f7f4c1f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f4b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38016/0 (socket says 172.21.15.32:38016) 2024-03-16T19:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.729+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/3399505055 learned_addr learned my addr 172.21.15.32:0/3399505055 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.733+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0cdb80 msgr2=0x7f7f4c1457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.733+0000 7f7f4b7fe640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0cdb80 0x7f7f4c1457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.733+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0c7830 msgr2=0x7f7f4c1f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.733+0000 7f7f4b7fe640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0c7830 0x7f7f4c1f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.733+0000 7f7f4b7fe640 1 -- 172.21.15.32:0/3399505055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44009a70 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.737+0000 7f7f4b7fe640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f4401ce40 tx=0x7f7f44019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440bf070 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f440aeac0 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440aee70 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c14a020 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c1415d0 con 0x7f7f4c0cdf60 2024-03-16T19:55:42.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:42.741+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c149d00 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.029+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 86353+0+0 (secure 0 0 0) 0x7f7f440b4050 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.029+0000 7f7f337fe640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 0x7f7f380692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:43.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.029+0000 7f7f4affd640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 0x7f7f380692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.029+0000 7f7f4affd640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 0x7f7f380692c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000f70 tx=0x7f7f3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.061+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7f440f0900 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.061+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7f440ecba0 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.173 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T19:55:43.201+0000 7efee40d27c0 -1 Falling back to public interface 2024-03-16T19:55:43.176 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T19:55:43.202+0000 7f8bb772b7c0 -1 Falling back to public interface 2024-03-16T19:55:43.177 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T19:55:43.202+0000 7fc39e6cb7c0 -1 Falling back to public interface 2024-03-16T19:55:43.185 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T19:55:43.213+0000 7fd4afdd47c0 -1 Falling back to public interface 2024-03-16T19:55:43.196 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T19:55:43.225+0000 7f8825b287c0 -1 Falling back to public interface 2024-03-16T19:55:43.196 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T19:55:43.218+0000 7eff176137c0 -1 Falling back to public interface 2024-03-16T19:55:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.233+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f4c068000 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.203 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T19:55:43.226+0000 7f9ba26a97c0 -1 Falling back to public interface 2024-03-16T19:55:43.220 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T19:55:43.249+0000 7fa0da1f57c0 -1 Falling back to public interface 2024-03-16T19:55:43.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.285+0000 7f7f337fe640 1 -- 172.21.15.32:0/3399505055 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f7f440b2090 con 0x7f7f4c0cdf60 2024-03-16T19:55:43.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:55:43.258 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:42.235009+0000","last_up_change":"0.000000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:55:43.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 msgr2=0x7f7f380692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 0x7f7f380692c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000f70 tx=0x7f7f3c009040 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 msgr2=0x7f7f4c0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f4401ce40 tx=0x7f7f44019700 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 shutdown_connections 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7f38066e00 0x7f7f380692c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f4c0cdf60 0x7f7f4c0c2e50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f4c0cdb80 0x7f7f4c1457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 --2- 172.21.15.32:0/3399505055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0c7830 0x7f7f4c1f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 >> 172.21.15.32:0/3399505055 conn(0x7f7f4c066ea0 msgr2=0x7f7f4c067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 shutdown_connections 2024-03-16T19:55:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:55:43.289+0000 7f7f5177f640 1 -- 172.21.15.32:0/3399505055 wait complete. 2024-03-16T19:55:43.278 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-16T19:55:43.278 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-16T19:55:44.084 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:55:44.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 Processor -- start 2024-03-16T19:55:44.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 -- start start 2024-03-16T19:55:44.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e006f9f0 0x7fc7e015b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7e0072830 0x7fc7e00c6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e00719e0 con 0x7fc7e0152060 2024-03-16T19:55:44.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e00c57d0 con 0x7fc7e0072830 2024-03-16T19:55:44.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7e5f52640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00c62b0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7df7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7e0152060 0x7fc7e0071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36266/0 (socket says 172.21.15.32:36266) 2024-03-16T19:55:44.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7df7fe640 1 -- 172.21.15.32:0/4017758866 learned_addr learned my addr 172.21.15.32:0/4017758866 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:55:44.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7dffff640 1 --2- 172.21.15.32:0/4017758866 >> v2:172.21.15.32:3300/0 conn(0x7fc7e0072830 0x7fc7e00c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.236+0000 7fc7deffd640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e006f9f0 0x7fc7e015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.240+0000 7fc7dffff640 1 -- 172.21.15.32:0/4017758866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e006f9f0 msgr2=0x7fc7e015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:44.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.240+0000 7fc7dffff640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e006f9f0 0x7fc7e015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.240+0000 7fc7dffff640 1 -- 172.21.15.32:0/4017758866 >> v1:172.21.15.32:6789/0 conn(0x7fc7e0152060 legacy=0x7fc7e0071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.240+0000 7fc7dffff640 1 -- 172.21.15.32:0/4017758866 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00c6c80 con 0x7fc7e0072830 2024-03-16T19:55:44.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.248+0000 7fc7dffff640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e0072830 0x7fc7e00c6740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009d90 tx=0x7fc7d801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f99c0282fda978 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:44.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7de7fc640 1 -- 172.21.15.32:0/4017758866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80a5970 con 0x7fc7e0072830 2024-03-16T19:55:44.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7de7fc640 1 -- 172.21.15.32:0/4017758866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d80a5ad0 con 0x7fc7e0072830 2024-03-16T19:55:44.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7de7fc640 1 -- 172.21.15.32:0/4017758866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80a5e60 con 0x7fc7e0072830 2024-03-16T19:55:44.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- 172.21.15.32:0/4017758866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e0072830 msgr2=0x7fc7e00c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e0072830 0x7fc7e00c6740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009d90 tx=0x7fc7d801c8b0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- 172.21.15.32:0/4017758866 shutdown_connections 2024-03-16T19:55:44.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e0072830 0x7fc7e00c6740 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/4017758866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e006f9f0 0x7fc7e015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- 172.21.15.32:0/4017758866 >> 172.21.15.32:0/4017758866 conn(0x7fc7e006afc0 msgr2=0x7fc7e006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:44.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- 172.21.15.32:0/4017758866 shutdown_connections 2024-03-16T19:55:44.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- 172.21.15.32:0/4017758866 wait complete. 2024-03-16T19:55:44.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 Processor -- start 2024-03-16T19:55:44.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.256+0000 7fc7e5f52640 1 -- start start 2024-03-16T19:55:44.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e0072830 0x7fc7e00c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7e0152060 0x7fc7e0144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01452f0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01f7460 con 0x7fc7e0152060 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7e5f52640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0152890 con 0x7fc7e0072830 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38034/0 (socket says 172.21.15.32:38034) 2024-03-16T19:55:44.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7df7fe640 1 -- 172.21.15.32:0/240091896 learned_addr learned my addr 172.21.15.32:0/240091896 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:55:44.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7deffd640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e0072830 0x7fc7e00c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.260+0000 7fc7dffff640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7e0152060 0x7fc7e0144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 -- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e0072830 msgr2=0x7fc7e00c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e0072830 0x7fc7e00c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 -- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7e0152060 msgr2=0x7fc7e0144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7e0152060 0x7fc7e0144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 -- 172.21.15.32:0/240091896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d8009a70 con 0x7fc7e006f9f0 2024-03-16T19:55:44.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.264+0000 7fc7df7fe640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c3a0 tx=0x7fc7d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:44.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0005cf0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d0005e50 con 0x7fc7e006f9f0 2024-03-16T19:55:44.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00072f0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e0144b10 con 0x7fc7e006f9f0 2024-03-16T19:55:44.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e0152dc0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.268+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e00c71d0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.288+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 86353+0+0 (secure 0 0 0) 0x7fc7d0007450 con 0x7fc7e006f9f0 2024-03-16T19:55:44.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.288+0000 7fc7d7fff640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 0x7fc7c00692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:44.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.288+0000 7fc7deffd640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 0x7fc7c00692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:44.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.292+0000 7fc7deffd640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 0x7fc7c00692c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0071400 tx=0x7fc7c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:44.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.292+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc7d0054f20 con 0x7fc7e006f9f0 2024-03-16T19:55:44.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.552+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7d00511c0 con 0x7fc7e006f9f0 2024-03-16T19:55:44.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.728+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7e0075440 con 0x7fc7e006f9f0 2024-03-16T19:55:44.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.764+0000 7fc7d7fff640 1 -- 172.21.15.32:0/240091896 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc7d0054880 con 0x7fc7e006f9f0 2024-03-16T19:55:44.739 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-16T19:55:44.739 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:42.235009+0000","last_up_change":"0.000000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:55:44.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 msgr2=0x7fc7c00692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 0x7fc7c00692c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0071400 tx=0x7fc7c8008040 comp rx=0 tx=0).stop 2024-03-16T19:55:44.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 msgr2=0x7fc7e01f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:44.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c3a0 tx=0x7fc7d000c860 comp rx=0 tx=0).stop 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 shutdown_connections 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7c0066e00 0x7fc7c00692c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7e0152060 0x7fc7e0144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e0072830 0x7fc7e00c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 --2- 172.21.15.32:0/240091896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e006f9f0 0x7fc7e01f6f20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 >> 172.21.15.32:0/240091896 conn(0x7fc7e006afc0 msgr2=0x7fc7e00c1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 shutdown_connections 2024-03-16T19:55:44.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:44.768+0000 7fc7e5f52640 1 -- 172.21.15.32:0/240091896 wait complete. 2024-03-16T19:55:44.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-16T19:55:46.723 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T19:55:46.748+0000 7efee40d27c0 -1 osd.1 0 log_to_monitors true 2024-03-16T19:55:47.017 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T19:55:47.042+0000 7fc39e6cb7c0 -1 osd.7 0 log_to_monitors true 2024-03-16T19:55:47.021 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T19:55:47.046+0000 7f8bb772b7c0 -1 osd.6 0 log_to_monitors true 2024-03-16T19:55:47.035 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T19:55:47.064+0000 7f8825b287c0 -1 osd.2 0 log_to_monitors true 2024-03-16T19:55:47.037 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T19:55:47.064+0000 7fd4afdd47c0 -1 osd.0 0 log_to_monitors true 2024-03-16T19:55:47.075 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T19:55:47.104+0000 7fa0da1f57c0 -1 osd.3 0 log_to_monitors true 2024-03-16T19:55:47.134 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T19:55:47.158+0000 7eff176137c0 -1 osd.5 0 log_to_monitors true 2024-03-16T19:55:47.134 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T19:55:47.158+0000 7f9ba26a97c0 -1 osd.4 0 log_to_monitors true 2024-03-16T19:55:49.133 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T19:55:49.160+0000 7efee088d640 -1 osd.1 0 waiting for initial osdmap 2024-03-16T19:55:49.320 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T19:55:49.348+0000 7efed7eb0640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:50.585 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T19:55:50.612+0000 7fa0d619e640 -1 osd.3 0 waiting for initial osdmap 2024-03-16T19:55:50.595 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T19:55:50.624+0000 7fd4abd7d640 -1 osd.0 0 waiting for initial osdmap 2024-03-16T19:55:50.613 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T19:55:50.640+0000 7f8821ad1640 -1 osd.2 0 waiting for initial osdmap 2024-03-16T19:55:50.659 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T19:55:50.688+0000 7fa0cdfd3640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:50.671 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T19:55:50.700+0000 7fd4a3bb2640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:50.679 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T19:55:50.708+0000 7f8819906640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:51.490 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T19:55:51.518+0000 7f8bb3ee6640 -1 osd.6 0 waiting for initial osdmap 2024-03-16T19:55:51.523 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T19:55:51.550+0000 7f8bab509640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:51.527 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T19:55:51.550+0000 7f9b9e652640 -1 osd.4 0 waiting for initial osdmap 2024-03-16T19:55:51.549 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T19:55:51.574+0000 7fc39a674640 -1 osd.7 0 waiting for initial osdmap 2024-03-16T19:55:51.550 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T19:55:51.574+0000 7eff135bc640 -1 osd.5 0 waiting for initial osdmap 2024-03-16T19:55:51.555 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T19:55:51.582+0000 7f9b96487640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:51.567 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:55:51.583 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T19:55:51.610+0000 7fc3924a9640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:51.583 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T19:55:51.610+0000 7eff0b3f1640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:55:51.694 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.720+0000 7f2c15315640 1 Processor -- start 2024-03-16T19:55:51.694 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.720+0000 7f2c15315640 1 -- start start 2024-03-16T19:55:51.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.724+0000 7f2c15315640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2c100c9300 0x7f2c10153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.724+0000 7f2c15315640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c101543e0 0x7f2c1015a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.724+0000 7f2c15315640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1015a240 con 0x7f2c100c8f20 2024-03-16T19:55:51.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.724+0000 7f2c15315640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c100cd120 con 0x7f2c100c9300 2024-03-16T19:55:51.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.724+0000 7f2c15315640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1015da40 con 0x7f2c101543e0 2024-03-16T19:55:51.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.748+0000 7f2c0effd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2c100c8f20 0x7f2c10072cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35506/0 (socket says 172.21.15.32:35506) 2024-03-16T19:55:51.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.748+0000 7f2c0effd640 1 -- 172.21.15.32:0/4194883416 learned_addr learned my addr 172.21.15.32:0/4194883416 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:55:51.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.748+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/4194883416 >> v2:172.21.15.32:3300/0 conn(0x7f2c100c9300 0x7f2c10153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:51.732 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.760+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/4194883416 >> v2:172.21.15.59:3300/0 conn(0x7f2c101543e0 msgr2=0x7f2c1015a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:51.733 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.760+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/4194883416 >> v2:172.21.15.59:3300/0 conn(0x7f2c101543e0 0x7f2c1015a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:51.733 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.760+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/4194883416 >> v1:172.21.15.32:6789/0 conn(0x7f2c100c8f20 legacy=0x7f2c10072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:51.733 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.760+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/4194883416 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1015dd20 con 0x7f2c100c9300 2024-03-16T19:55:51.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.768+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/4194883416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c10153ea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2c100c4520 tx=0x7f2c0001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdd6e5065766d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:51.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c0dffb640 1 -- 172.21.15.32:0/4194883416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000a8970 con 0x7f2c100c9300 2024-03-16T19:55:51.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c0dffb640 1 -- 172.21.15.32:0/4194883416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c000a8ad0 con 0x7f2c100c9300 2024-03-16T19:55:51.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c0dffb640 1 -- 172.21.15.32:0/4194883416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000a8e60 con 0x7f2c100c9300 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 -- 172.21.15.32:0/4194883416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 msgr2=0x7f2c10153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 --2- 172.21.15.32:0/4194883416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c10153ea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2c100c4520 tx=0x7f2c0001ec70 comp rx=0 tx=0).stop 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 -- 172.21.15.32:0/4194883416 shutdown_connections 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 --2- 172.21.15.32:0/4194883416 >> v2:172.21.15.59:3300/0 conn(0x7f2c101543e0 0x7f2c1015a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 --2- 172.21.15.32:0/4194883416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c10153ea0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 -- 172.21.15.32:0/4194883416 >> 172.21.15.32:0/4194883416 conn(0x7f2c1006fea0 msgr2=0x7f2c100702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 -- 172.21.15.32:0/4194883416 shutdown_connections 2024-03-16T19:55:51.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.780+0000 7f2c15315640 1 -- 172.21.15.32:0/4194883416 wait complete. 2024-03-16T19:55:51.755 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 Processor -- start 2024-03-16T19:55:51.755 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 -- start start 2024-03-16T19:55:51.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c100c8f20 0x7f2c101f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c101543e0 0x7f2c1014d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c10153320 con 0x7f2c100c9300 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c101f4df0 con 0x7f2c101543e0 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c15315640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c100c5080 con 0x7f2c100c8f20 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c0e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c0e7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48654/0 (socket says 172.21.15.32:48654) 2024-03-16T19:55:51.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.784+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/1681912999 learned_addr learned my addr 172.21.15.32:0/1681912999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:55:51.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.788+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c100c8f20 msgr2=0x7f2c101f48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:51.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.788+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c100c8f20 0x7f2c101f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:51.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.788+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c101543e0 msgr2=0x7f2c1014d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:51.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.788+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c101543e0 0x7f2c1014d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:51.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.788+0000 7f2c0e7fc640 1 -- 172.21.15.32:0/1681912999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c00009420 con 0x7f2c100c9300 2024-03-16T19:55:51.764 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.792+0000 7f2c0e7fc640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c0001c8a0 tx=0x7f2c0001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:51.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000a8910 con 0x7f2c100c9300 2024-03-16T19:55:51.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c000b9410 con 0x7f2c100c9300 2024-03-16T19:55:51.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000b97c0 con 0x7f2c100c9300 2024-03-16T19:55:51.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1014d050 con 0x7f2c100c9300 2024-03-16T19:55:51.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c100c4b00 con 0x7f2c100c9300 2024-03-16T19:55:51.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.796+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c10068510 con 0x7f2c100c9300 2024-03-16T19:55:51.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.820+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 86353+0+0 (secure 0 0 0) 0x7f2c000b7040 con 0x7f2c100c9300 2024-03-16T19:55:51.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.824+0000 7f2bf77fe640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 0x7f2bfc0692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:51.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.824+0000 7f2c0effd640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 0x7f2bfc0692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:51.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:51.824+0000 7f2c0effd640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 0x7f2bfc0692c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c04000f70 tx=0x7f2c04007a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:55:52.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.132+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f2c000f4230 con 0x7f2c100c9300 2024-03-16T19:55:52.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.132+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2c000efdd0 con 0x7f2c100c9300 2024-03-16T19:55:52.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.316+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c1014ce70 con 0x7f2c100c9300 2024-03-16T19:55:52.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.496+0000 7f2bf77fe640 1 -- 172.21.15.32:0/1681912999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9000 (secure 0 0 0) 0x7f2c000b5020 con 0x7f2c100c9300 2024-03-16T19:55:52.470 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-16T19:55:52.471 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":15,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:51.709701+0000","last_up_change":"2024-03-16T19:55:51.709701+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:55:52.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 msgr2=0x7f2bfc0692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:52.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 0x7f2bfc0692c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c04000f70 tx=0x7f2c04007a90 comp rx=0 tx=0).stop 2024-03-16T19:55:52.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 msgr2=0x7f2c100cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:52.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c0001c8a0 tx=0x7f2c0001ba10 comp rx=0 tx=0).stop 2024-03-16T19:55:52.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 shutdown_connections 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bfc066e00 0x7f2bfc0692c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c101543e0 0x7f2c1014d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c100c9300 0x7f2c100cb810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 --2- 172.21.15.32:0/1681912999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c100c8f20 0x7f2c101f48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 >> 172.21.15.32:0/1681912999 conn(0x7f2c1006fea0 msgr2=0x7f2c10072cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 shutdown_connections 2024-03-16T19:55:52.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:52.500+0000 7f2c15315640 1 -- 172.21.15.32:0/1681912999 wait complete. 2024-03-16T19:55:52.493 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-03-16T19:55:59.299 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:55:59.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 Processor -- start 2024-03-16T19:55:59.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 -- start start 2024-03-16T19:55:59.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd68406fae0 0x7fd68415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:59.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd68415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:55:59.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68406c250 con 0x7fd684154470 2024-03-16T19:55:59.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68406c530 con 0x7fd68406fae0 2024-03-16T19:55:59.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd68b521640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68406ed20 con 0x7fd6840cae80 2024-03-16T19:55:59.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd688abc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd68406fae0 0x7fd68415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:59.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd6892bd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd684154470 0x7fd6840c2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44434/0 (socket says 172.21.15.32:44434) 2024-03-16T19:55:59.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd6892bd640 1 -- 172.21.15.32:0/938081739 learned_addr learned my addr 172.21.15.32:0/938081739 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:55:59.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.444+0000 7fd689abe640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd68415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:55:59.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.448+0000 7fd688abc640 1 -- 172.21.15.32:0/938081739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 msgr2=0x7fd68415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:55:59.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.448+0000 7fd688abc640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd68415dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:55:59.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.448+0000 7fd688abc640 1 -- 172.21.15.32:0/938081739 >> v1:172.21.15.32:6789/0 conn(0x7fd684154470 legacy=0x7fd6840c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:55:59.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.448+0000 7fd688abc640 1 -- 172.21.15.32:0/938081739 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68406f000 con 0x7fd68406fae0 2024-03-16T19:55:59.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:55:59.452+0000 7fd688abc640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68406fae0 0x7fd68415d6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6740097c0 tx=0x7fd67401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621b306a45a3e984 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:02.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.004+0000 7fd67bfff640 1 -- 172.21.15.32:0/938081739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740ab670 con 0x7fd68406fae0 2024-03-16T19:56:02.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.004+0000 7fd67bfff640 1 -- 172.21.15.32:0/938081739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6740ab7f0 con 0x7fd68406fae0 2024-03-16T19:56:02.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.004+0000 7fd67bfff640 1 -- 172.21.15.32:0/938081739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740abbe0 con 0x7fd68406fae0 2024-03-16T19:56:02.979 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- 172.21.15.32:0/938081739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68406fae0 msgr2=0x7fd68415d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68406fae0 0x7fd68415d6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6740097c0 tx=0x7fd67401c470 comp rx=0 tx=0).stop 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- 172.21.15.32:0/938081739 shutdown_connections 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd68415dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 --2- 172.21.15.32:0/938081739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68406fae0 0x7fd68415d6c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- 172.21.15.32:0/938081739 >> 172.21.15.32:0/938081739 conn(0x7fd684070fa0 msgr2=0x7fd6840713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- 172.21.15.32:0/938081739 shutdown_connections 2024-03-16T19:56:02.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- 172.21.15.32:0/938081739 wait complete. 2024-03-16T19:56:02.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 Processor -- start 2024-03-16T19:56:02.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.008+0000 7fd68b521640 1 -- start start 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd684141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd684154470 0x7fd684142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684158b10 con 0x7fd684154470 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6841429d0 con 0x7fd68406fae0 2024-03-16T19:56:02.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd68b521640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd684159100 con 0x7fd6840cae80 2024-03-16T19:56:02.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.012+0000 7fd6892bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:02.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.016+0000 7fd6892bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:39726/0 (socket says 172.21.15.32:39726) 2024-03-16T19:56:02.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.016+0000 7fd6892bd640 1 -- 172.21.15.32:0/4199472684 learned_addr learned my addr 172.21.15.32:0/4199472684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:02.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.016+0000 7fd688abc640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd684141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:03.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.028+0000 7fd6892bd640 1 -- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 msgr2=0x7fd684141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:03.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.028+0000 7fd6892bd640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd684141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:03.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.028+0000 7fd6892bd640 1 -- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd684154470 msgr2=0x7fd684142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:03.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.028+0000 7fd6892bd640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd684154470 0x7fd684142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:03.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.028+0000 7fd6892bd640 1 -- 172.21.15.32:0/4199472684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674009420 con 0x7fd68406fae0 2024-03-16T19:56:03.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.036+0000 7fd6892bd640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd68000caa0 tx=0x7fd68000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:03.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.044+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68000aba0 con 0x7fd68406fae0 2024-03-16T19:56:03.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.044+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6841ec0e0 con 0x7fd68406fae0 2024-03-16T19:56:03.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.044+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6841f6d60 con 0x7fd68406fae0 2024-03-16T19:56:03.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.044+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd684075440 con 0x7fd68406fae0 2024-03-16T19:56:03.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.048+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680001b40 con 0x7fd68406fae0 2024-03-16T19:56:03.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.048+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68001c440 con 0x7fd68406fae0 2024-03-16T19:56:03.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.404+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd680012400 con 0x7fd68406fae0 2024-03-16T19:56:03.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.404+0000 7fd679ffb640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 0x7fd668069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:03.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.404+0000 7fd688abc640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 0x7fd668069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:03.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.408+0000 7fd688abc640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 0x7fd668069300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6740af000 tx=0x7fd6740a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:03.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.408+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd680055cd0 con 0x7fd68406fae0 2024-03-16T19:56:03.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.424+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd680055aa0 con 0x7fd68406fae0 2024-03-16T19:56:03.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:03.604+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6840c8890 con 0x7fd68406fae0 2024-03-16T19:56:09.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.680+0000 7fd679ffb640 1 -- 172.21.15.32:0/4199472684 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12824 (secure 0 0 0) 0x7fd680021070 con 0x7fd68406fae0 2024-03-16T19:56:09.654 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-16T19:56:09.654 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":18,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:56.041430+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 msgr2=0x7fd668069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 0x7fd668069300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6740af000 tx=0x7fd6740a7000 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 msgr2=0x7fd684150e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd68000caa0 tx=0x7fd68000cf60 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 shutdown_connections 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd668066e40 0x7fd668069300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd684154470 0x7fd684142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd6840cae80 0x7fd684141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 --2- 172.21.15.32:0/4199472684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd68406fae0 0x7fd684150e70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 >> 172.21.15.32:0/4199472684 conn(0x7fd684070fa0 msgr2=0x7fd6840c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 shutdown_connections 2024-03-16T19:56:09.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-16T19:56:09.684+0000 7fd68b521640 1 -- 172.21.15.32:0/4199472684 wait complete. 2024-03-16T19:56:09.688 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-16T19:56:09.688 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-16T19:56:09.688 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.688 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-16T19:56:09.688 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-16T19:56:09.690 INFO:tasks.ceph.mds.a:Started 2024-03-16T19:56:09.691 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.691 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-16T19:56:09.691 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-16T19:56:09.693 INFO:tasks.ceph.mds.c:Started 2024-03-16T19:56:09.693 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.693 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-16T19:56:09.693 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-16T19:56:09.695 INFO:tasks.ceph.mds.e:Started 2024-03-16T19:56:09.695 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.696 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-16T19:56:09.696 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-16T19:56:09.698 INFO:tasks.ceph.mds.b:Started 2024-03-16T19:56:09.699 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.699 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-16T19:56:09.699 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-16T19:56:09.701 INFO:tasks.ceph.mds.d:Started 2024-03-16T19:56:09.701 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-16T19:56:09.701 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-16T19:56:09.702 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-16T19:56:09.704 INFO:tasks.ceph.mds.f:Started 2024-03-16T19:56:09.704 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-16T19:56:09.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 Processor -- start 2024-03-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 -- start start 2024-03-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef38072d50 0x7fef3815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef380c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380c7c90 con 0x7fef38151ef0 2024-03-16T19:56:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef380c70d0 con 0x7fef38072d50 2024-03-16T19:56:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef3c943640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef38072a00 con 0x7fef380c7800 2024-03-16T19:56:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef2ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef38072d50 0x7fef3815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:09.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef2ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef38072d50 0x7fef3815b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46140/0 (socket says 172.21.15.32:46140) 2024-03-16T19:56:09.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.916+0000 7fef2ffff640 1 -- 172.21.15.32:0/1166847657 learned_addr learned my addr 172.21.15.32:0/1166847657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.920+0000 7fef2ffff640 1 -- 172.21.15.32:0/1166847657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 msgr2=0x7fef380c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:09.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.920+0000 7fef2ffff640 1 --2- 172.21.15.32:0/1166847657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef380c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.920+0000 7fef2ffff640 1 -- 172.21.15.32:0/1166847657 >> v1:172.21.15.32:6789/0 conn(0x7fef38151ef0 legacy=0x7fef380722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:09.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.920+0000 7fef2ffff640 1 -- 172.21.15.32:0/1166847657 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef380c96b0 con 0x7fef38072d50 2024-03-16T19:56:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.928+0000 7fef2ffff640 1 --2- 172.21.15.32:0/1166847657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38072d50 0x7fef3815b2d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef20009870 tx=0x7fef2001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6b61b4080c0abf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef35d9b640 1 -- 172.21.15.32:0/1166847657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef200a89d0 con 0x7fef38072d50 2024-03-16T19:56:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef35d9b640 1 -- 172.21.15.32:0/1166847657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef200a8b30 con 0x7fef38072d50 2024-03-16T19:56:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef35d9b640 1 -- 172.21.15.32:0/1166847657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef200b0440 con 0x7fef38072d50 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- 172.21.15.32:0/1166847657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38072d50 msgr2=0x7fef3815b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 --2- 172.21.15.32:0/1166847657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38072d50 0x7fef3815b2d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef20009870 tx=0x7fef2001ed10 comp rx=0 tx=0).stop 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- 172.21.15.32:0/1166847657 shutdown_connections 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 --2- 172.21.15.32:0/1166847657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef380c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 --2- 172.21.15.32:0/1166847657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38072d50 0x7fef3815b2d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- 172.21.15.32:0/1166847657 >> 172.21.15.32:0/1166847657 conn(0x7fef3806e6c0 msgr2=0x7fef3806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- 172.21.15.32:0/1166847657 shutdown_connections 2024-03-16T19:56:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- 172.21.15.32:0/1166847657 wait complete. 2024-03-16T19:56:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 Processor -- start 2024-03-16T19:56:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.936+0000 7fef3c943640 1 -- start start 2024-03-16T19:56:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38072d50 0x7fef380cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef381e3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381ee4f0 con 0x7fef38151ef0 2024-03-16T19:56:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef380cc8d0 con 0x7fef38072d50 2024-03-16T19:56:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3c943640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef381e3850 con 0x7fef380c7800 2024-03-16T19:56:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef36d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef36d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46152/0 (socket says 172.21.15.32:46152) 2024-03-16T19:56:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef36d9d640 1 -- 172.21.15.32:0/2810613667 learned_addr learned my addr 172.21.15.32:0/2810613667 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.940+0000 7fef3659c640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38072d50 0x7fef380cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:09.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 -- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 msgr2=0x7fef381e3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef381e3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 -- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38072d50 msgr2=0x7fef380cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38072d50 0x7fef380cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 -- 172.21.15.32:0/2810613667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef200094d0 con 0x7fef38151ef0 2024-03-16T19:56:09.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.948+0000 7fef36d9d640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b970 tx=0x7fef3000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:09.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3000cb40 con 0x7fef38151ef0 2024-03-16T19:56:09.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3000ccc0 con 0x7fef38151ef0 2024-03-16T19:56:09.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30014680 con 0x7fef38151ef0 2024-03-16T19:56:09.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef381f0cd0 con 0x7fef38151ef0 2024-03-16T19:56:09.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef381ed870 con 0x7fef38151ef0 2024-03-16T19:56:09.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.956+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef381ee630 con 0x7fef38151ef0 2024-03-16T19:56:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.992+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef3001d3b0 con 0x7fef38151ef0 2024-03-16T19:56:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.992+0000 7fef2effd640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 0x7fef18069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.992+0000 7fef3659c640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 0x7fef18069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:09.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.992+0000 7fef3659c640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 0x7fef18069300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef280098e0 tx=0x7fef28009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:09.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:09.996+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fef3000ce30 con 0x7fef38151ef0 2024-03-16T19:56:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.292+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fef30051a60 con 0x7fef38151ef0 2024-03-16T19:56:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.528+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef381ee630 con 0x7fef38151ef0 2024-03-16T19:56:10.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.620+0000 7fef2effd640 1 -- 172.21.15.32:0/2810613667 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12824 (secure 0 0 0) 0x7fef381ee630 con 0x7fef38151ef0 2024-03-16T19:56:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:10.595 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:56.041430+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 msgr2=0x7fef18069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 0x7fef18069300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef280098e0 tx=0x7fef28009060 comp rx=0 tx=0).stop 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 msgr2=0x7fef380cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b970 tx=0x7fef3000be30 comp rx=0 tx=0).stop 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 shutdown_connections 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef18066e40 0x7fef18069300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef38151ef0 0x7fef380cced0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef380c7800 0x7fef381e3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 --2- 172.21.15.32:0/2810613667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38072d50 0x7fef380cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 >> 172.21.15.32:0/2810613667 conn(0x7fef3806e6c0 msgr2=0x7fef381502b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 shutdown_connections 2024-03-16T19:56:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.624+0000 7fef3c943640 1 -- 172.21.15.32:0/2810613667 wait complete. 2024-03-16T19:56:10.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:56:10.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:56:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.752+0000 7f1dd0254640 1 Processor -- start 2024-03-16T19:56:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 -- start start 2024-03-16T19:56:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dc806cae0 0x7f1dc80c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc806c460 con 0x7f1dc80c3b80 2024-03-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc80c2e10 con 0x7f1dc806cae0 2024-03-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dd0254640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc8154630 con 0x7f1dc8154250 2024-03-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dcd7ef640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dc806cae0 0x7f1dc80c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dcdff0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dcd7ef640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dc806cae0 0x7f1dc80c3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46166/0 (socket says 172.21.15.32:46166) 2024-03-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.756+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/4159036488 learned_addr learned my addr 172.21.15.32:0/4159036488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.760+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/4159036488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 msgr2=0x7f1dc815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.760+0000 7f1dcd7ef640 1 --2- 172.21.15.32:0/4159036488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.760+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/4159036488 >> v1:172.21.15.32:6789/0 conn(0x7f1dc80c3b80 legacy=0x7f1dc80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.760+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/4159036488 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc815a360 con 0x7f1dc806cae0 2024-03-16T19:56:10.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.764+0000 7f1dcd7ef640 1 --2- 172.21.15.32:0/4159036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc806cae0 0x7f1dc80c3640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009870 tx=0x7f1db801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b9f1650463f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.768+0000 7f1dccfee640 1 -- 172.21.15.32:0/4159036488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80a8a50 con 0x7f1dc806cae0 2024-03-16T19:56:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dccfee640 1 -- 172.21.15.32:0/4159036488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80a8bb0 con 0x7f1dc806cae0 2024-03-16T19:56:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dccfee640 1 -- 172.21.15.32:0/4159036488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b0440 con 0x7f1dc806cae0 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- 172.21.15.32:0/4159036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc806cae0 msgr2=0x7f1dc80c3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- 172.21.15.32:0/4159036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc806cae0 0x7f1dc80c3640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009870 tx=0x7f1db801ed50 comp rx=0 tx=0).stop 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- 172.21.15.32:0/4159036488 shutdown_connections 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- 172.21.15.32:0/4159036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc806cae0 0x7f1dc80c3640 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- 172.21.15.32:0/4159036488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- 172.21.15.32:0/4159036488 >> 172.21.15.32:0/4159036488 conn(0x7f1dc8070f90 msgr2=0x7f1dc80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- 172.21.15.32:0/4159036488 shutdown_connections 2024-03-16T19:56:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- 172.21.15.32:0/4159036488 wait complete. 2024-03-16T19:56:10.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 Processor -- start 2024-03-16T19:56:10.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- start start 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc806cae0 0x7f1dc80cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc81ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc81f3a40 con 0x7f1dc80c3b80 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc80ccc60 con 0x7f1dc806cae0 2024-03-16T19:56:10.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dd0254640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc81f2e90 con 0x7f1dc8154250 2024-03-16T19:56:10.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.772+0000 7f1dcd7ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.776+0000 7f1dce7f1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc81ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.776+0000 7f1dcdff0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc806cae0 0x7f1dc80cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.776+0000 7f1dcd7ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46182/0 (socket says 172.21.15.32:46182) 2024-03-16T19:56:10.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.776+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/2581403364 learned_addr learned my addr 172.21.15.32:0/2581403364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.780+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 msgr2=0x7f1dc81ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.780+0000 7f1dcd7ef640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc81ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.780+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc806cae0 msgr2=0x7f1dc80cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.780+0000 7f1dcd7ef640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc806cae0 0x7f1dc80cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.780+0000 7f1dcd7ef640 1 -- 172.21.15.32:0/2581403364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80094d0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.784+0000 7f1dcd7ef640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1db80b3040 tx=0x7f1db801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.788+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b0dd0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.788+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80bb460 con 0x7f1dc80c3b80 2024-03-16T19:56:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.788+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc80ccec0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.788+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80bb810 con 0x7f1dc80c3b80 2024-03-16T19:56:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.788+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc81f33f0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.792+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc81f3bf0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.816+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1db80b83f0 con 0x7f1dc80c3b80 2024-03-16T19:56:10.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.816+0000 7f1dbe7fc640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 0x7f1dac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:10.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.816+0000 7f1dcdff0640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 0x7f1dac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:10.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:10.816+0000 7f1dcdff0640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 0x7f1dac069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4009ac0 tx=0x7f1dc4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.084+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1db80f3250 con 0x7f1dc80c3b80 2024-03-16T19:56:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.084+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1db80ef4f0 con 0x7f1dc80c3b80 2024-03-16T19:56:11.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.264+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1dc81f3bf0 con 0x7f1dc80c3b80 2024-03-16T19:56:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.300+0000 7f1dbe7fc640 1 -- 172.21.15.32:0/2581403364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1dc81f3bf0 con 0x7f1dc80c3b80 2024-03-16T19:56:11.271 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:56:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.300+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 msgr2=0x7f1dac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.300+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 0x7f1dac069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4009ac0 tx=0x7f1dc4009190 comp rx=0 tx=0).stop 2024-03-16T19:56:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.300+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 msgr2=0x7f1dc814ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.300+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1db80b3040 tx=0x7f1db801baf0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 shutdown_connections 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1dac066e90 0x7f1dac069350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc8154250 0x7f1dc81ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc80c3b80 0x7f1dc814ec00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 --2- 172.21.15.32:0/2581403364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc806cae0 0x7f1dc80cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 >> 172.21.15.32:0/2581403364 conn(0x7f1dc8070f90 msgr2=0x7f1dc80ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 shutdown_connections 2024-03-16T19:56:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.304+0000 7f1dd0254640 1 -- 172.21.15.32:0/2581403364 wait complete. 2024-03-16T19:56:11.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 Processor -- start 2024-03-16T19:56:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 -- start start 2024-03-16T19:56:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58b814ea20 0x7f58b806ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58b80c9e50 0x7f58b80c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b8072e80 con 0x7f58b80c9860 2024-03-16T19:56:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b80c8b10 con 0x7f58b814ea20 2024-03-16T19:56:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bf8af640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b8157d60 con 0x7f58b80c9e50 2024-03-16T19:56:11.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bd64b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58b814ea20 0x7f58b806ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bd64b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58b814ea20 0x7f58b806ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46188/0 (socket says 172.21.15.32:46188) 2024-03-16T19:56:11.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bd64b640 1 -- 172.21.15.32:0/1101160141 learned_addr learned my addr 172.21.15.32:0/1101160141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:11.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.428+0000 7f58bce4a640 1 --2- 172.21.15.32:0/1101160141 >> v2:172.21.15.59:3300/0 conn(0x7f58b80c9e50 0x7f58b80c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.432+0000 7f58bd64b640 1 -- 172.21.15.32:0/1101160141 >> v2:172.21.15.59:3300/0 conn(0x7f58b80c9e50 msgr2=0x7f58b80c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:11.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.432+0000 7f58bd64b640 1 --2- 172.21.15.32:0/1101160141 >> v2:172.21.15.59:3300/0 conn(0x7f58b80c9e50 0x7f58b80c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.432+0000 7f58bd64b640 1 -- 172.21.15.32:0/1101160141 >> v1:172.21.15.59:6789/0 conn(0x7f58b80c9860 legacy=0x7f58b8158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.432+0000 7f58bd64b640 1 -- 172.21.15.32:0/1101160141 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b8072390 con 0x7f58b814ea20 2024-03-16T19:56:11.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.436+0000 7f58bd64b640 1 --2- 172.21.15.32:0/1101160141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b814ea20 0x7f58b806ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58b806cac0 tx=0x7f58b401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c740484f7f07ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:11.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58affff640 1 -- 172.21.15.32:0/1101160141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40aa650 con 0x7f58b814ea20 2024-03-16T19:56:11.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58affff640 1 -- 172.21.15.32:0/1101160141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b40aa7b0 con 0x7f58b814ea20 2024-03-16T19:56:11.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58affff640 1 -- 172.21.15.32:0/1101160141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40aab40 con 0x7f58b814ea20 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- 172.21.15.32:0/1101160141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b814ea20 msgr2=0x7f58b806ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1101160141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b814ea20 0x7f58b806ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58b806cac0 tx=0x7f58b401c4f0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- 172.21.15.32:0/1101160141 shutdown_connections 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1101160141 >> v2:172.21.15.59:3300/0 conn(0x7f58b80c9e50 0x7f58b80c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1101160141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b814ea20 0x7f58b806ba60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- 172.21.15.32:0/1101160141 >> 172.21.15.32:0/1101160141 conn(0x7f58b806e6c0 msgr2=0x7f58b806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:11.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- 172.21.15.32:0/1101160141 shutdown_connections 2024-03-16T19:56:11.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- 172.21.15.32:0/1101160141 wait complete. 2024-03-16T19:56:11.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 Processor -- start 2024-03-16T19:56:11.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.444+0000 7f58bf8af640 1 -- start start 2024-03-16T19:56:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b80c9860 0x7f58b813afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 0x7f58b8143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 0x7f58b8156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b814c300 con 0x7f58b80c9e50 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b814a9a0 con 0x7f58b80c9860 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bf8af640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b81f3500 con 0x7f58b814ea20 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bce4a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 0x7f58b8143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bde4c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 0x7f58b8156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bce4a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 0x7f58b8143050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46190/0 (socket says 172.21.15.32:46190) 2024-03-16T19:56:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bce4a640 1 -- 172.21.15.32:0/1299503672 learned_addr learned my addr 172.21.15.32:0/1299503672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.448+0000 7f58bd64b640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b80c9860 0x7f58b813afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.452+0000 7f58bde4c640 1 -- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b80c9860 msgr2=0x7f58b813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.452+0000 7f58bde4c640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b80c9860 0x7f58b813afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.452+0000 7f58bde4c640 1 -- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 msgr2=0x7f58b8143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.452+0000 7f58bde4c640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 0x7f58b8143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.452+0000 7f58bde4c640 1 -- 172.21.15.32:0/1299503672 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b40094d0 con 0x7f58b814ea20 2024-03-16T19:56:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.456+0000 7f58bde4c640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 0x7f58b8156a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58b000c730 tx=0x7f58b000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:11.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.460+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0006d60 con 0x7f58b814ea20 2024-03-16T19:56:11.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.460+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b814abf0 con 0x7f58b814ea20 2024-03-16T19:56:11.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.460+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b814b1b0 con 0x7f58b814ea20 2024-03-16T19:56:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.460+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b81f3720 con 0x7f58b814ea20 2024-03-16T19:56:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.464+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b0001680 con 0x7f58b814ea20 2024-03-16T19:56:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.464+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0003b80 con 0x7f58b814ea20 2024-03-16T19:56:11.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:56:11.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:56:11.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.928+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58b0003ce0 con 0x7f58b814ea20 2024-03-16T19:56:11.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.928+0000 7f58adffb640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 0x7f589c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:11.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.928+0000 7f58bd64b640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 0x7f589c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.932+0000 7f58bd64b640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 0x7f589c069330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f58b40ae000 tx=0x7f58b40b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.936+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f58b0055bb0 con 0x7f58b814ea20 2024-03-16T19:56:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:11.956+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f58b0051e50 con 0x7f58b814ea20 2024-03-16T19:56:12.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:12.168+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58b81f3720 con 0x7f58b814ea20 2024-03-16T19:56:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58adffb640 1 -- 172.21.15.32:0/1299503672 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f58b81f3720 con 0x7f58b814ea20 2024-03-16T19:56:14.504 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:14.504 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T19:56:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-03-16T19:56:14.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 msgr2=0x7f589c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 0x7f589c069330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f58b40ae000 tx=0x7f58b40b5040 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 msgr2=0x7f58b8156a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 0x7f58b8156a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58b000c730 tx=0x7f58b000cbf0 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 shutdown_connections 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f589c066e70 0x7f589c069330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b814ea20 0x7f58b8156a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80c9e50 0x7f58b8143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 --2- 172.21.15.32:0/1299503672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b80c9860 0x7f58b813afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 >> 172.21.15.32:0/1299503672 conn(0x7f58b806e6c0 msgr2=0x7f58b81482d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 shutdown_connections 2024-03-16T19:56:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:14.532+0000 7f58bf8af640 1 -- 172.21.15.32:0/1299503672 wait complete. 2024-03-16T19:56:16.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.688+0000 7f8f23812640 1 Processor -- start 2024-03-16T19:56:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f23812640 1 -- start start 2024-03-16T19:56:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f23812640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f1c0c8550 0x7f8f1c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f23812640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c15ab60 con 0x7f8f1c1544a0 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f23812640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c159fb0 con 0x7f8f1c1540c0 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f23812640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1c06fb80 con 0x7f8f1c0c8550 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f21daf640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f1c0c8550 0x7f8f1c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f215ae640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8f1c1540c0 0x7f8f1c06f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:56044/0 (socket says 172.21.15.32:56044) 2024-03-16T19:56:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.692+0000 7f8f215ae640 1 -- 172.21.15.32:0/4030921695 learned_addr learned my addr 172.21.15.32:0/4030921695 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.696+0000 7f8f21daf640 1 -- 172.21.15.32:0/4030921695 >> v1:172.21.15.59:6789/0 conn(0x7f8f1c1540c0 legacy=0x7f8f1c06f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.696+0000 7f8f21daf640 1 -- 172.21.15.32:0/4030921695 >> v1:172.21.15.32:6789/0 conn(0x7f8f1c1544a0 legacy=0x7f8f1c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.696+0000 7f8f21daf640 1 -- 172.21.15.32:0/4030921695 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c15a230 con 0x7f8f1c0c8550 2024-03-16T19:56:16.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.700+0000 7f8f21daf640 1 --2- 172.21.15.32:0/4030921695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0cd830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009d90 tx=0x7f8f1401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef232e742141c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f13fff640 1 -- 172.21.15.32:0/4030921695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140a5810 con 0x7f8f1c0c8550 2024-03-16T19:56:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f13fff640 1 -- 172.21.15.32:0/4030921695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140a5970 con 0x7f8f1c0c8550 2024-03-16T19:56:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f13fff640 1 -- 172.21.15.32:0/4030921695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140a5d00 con 0x7f8f1c0c8550 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 -- 172.21.15.32:0/4030921695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c0c8550 msgr2=0x7f8f1c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 --2- 172.21.15.32:0/4030921695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0cd830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009d90 tx=0x7f8f1401c970 comp rx=0 tx=0).stop 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 -- 172.21.15.32:0/4030921695 shutdown_connections 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 --2- 172.21.15.32:0/4030921695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0cd830 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 -- 172.21.15.32:0/4030921695 >> 172.21.15.32:0/4030921695 conn(0x7f8f1c066ea0 msgr2=0x7f8f1c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:16.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 -- 172.21.15.32:0/4030921695 shutdown_connections 2024-03-16T19:56:16.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 -- 172.21.15.32:0/4030921695 wait complete. 2024-03-16T19:56:16.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.704+0000 7f8f23812640 1 Processor -- start 2024-03-16T19:56:16.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 -- start start 2024-03-16T19:56:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f1c1540c0 0x7f8f1c151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c155870 con 0x7f8f1c1544a0 2024-03-16T19:56:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c156400 con 0x7f8f1c0c8550 2024-03-16T19:56:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f23812640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0c5270 con 0x7f8f1c1540c0 2024-03-16T19:56:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f21daf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f21daf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46226/0 (socket says 172.21.15.32:46226) 2024-03-16T19:56:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f215ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f21daf640 1 -- 172.21.15.32:0/1775128218 learned_addr learned my addr 172.21.15.32:0/1775128218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:16.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.708+0000 7f8f20dad640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f1c1540c0 0x7f8f1c151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.712+0000 7f8f21daf640 1 -- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f1c1540c0 msgr2=0x7f8f1c151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.712+0000 7f8f21daf640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f1c1540c0 0x7f8f1c151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.712+0000 7f8f21daf640 1 -- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f1c0c8550 msgr2=0x7f8f1c0c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.712+0000 7f8f21daf640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.712+0000 7f8f21daf640 1 -- 172.21.15.32:0/1775128218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f14009a70 con 0x7f8f1c1544a0 2024-03-16T19:56:16.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.716+0000 7f8f21daf640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f1401a5a0 tx=0x7f8f14019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:16.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.720+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140ad850 con 0x7f8f1c1544a0 2024-03-16T19:56:16.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.720+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140ad9b0 con 0x7f8f1c1544a0 2024-03-16T19:56:16.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.720+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c0c7ae0 con 0x7f8f1c1544a0 2024-03-16T19:56:16.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.720+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140add40 con 0x7f8f1c1544a0 2024-03-16T19:56:16.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.720+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c156c60 con 0x7f8f1c1544a0 2024-03-16T19:56:16.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:16.724+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c156df0 con 0x7f8f1c1544a0 2024-03-16T19:56:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.044+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8f140b4070 con 0x7f8f1c1544a0 2024-03-16T19:56:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.044+0000 7f8f11ffb640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 0x7f8f00069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.044+0000 7f8f215ae640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 0x7f8f00069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:17.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.044+0000 7f8f215ae640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 0x7f8f00069350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f18005890 tx=0x7f8f18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:17.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.064+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8f140f0cc0 con 0x7f8f1c1544a0 2024-03-16T19:56:17.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.064+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8f140ecf60 con 0x7f8f1c1544a0 2024-03-16T19:56:17.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.328+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f1c0c7ae0 con 0x7f8f1c1544a0 2024-03-16T19:56:17.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.340+0000 7f8f11ffb640 1 -- 172.21.15.32:0/1775128218 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f8f140b2230 con 0x7f8f1c1544a0 2024-03-16T19:56:17.314 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:17.314 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T19:56:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-03-16T19:56:17.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 msgr2=0x7f8f00069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:17.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 0x7f8f00069350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f18005890 tx=0x7f8f18009040 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 msgr2=0x7f8f1c1570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f1401a5a0 tx=0x7f8f14019710 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 shutdown_connections 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f00066e90 0x7f8f00069350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f1c1544a0 0x7f8f1c1570f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f1c1540c0 0x7f8f1c151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 --2- 172.21.15.32:0/1775128218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f1c0c8550 0x7f8f1c0c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 >> 172.21.15.32:0/1775128218 conn(0x7f8f1c066ea0 msgr2=0x7f8f1c1523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 shutdown_connections 2024-03-16T19:56:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:17.344+0000 7f8f23812640 1 -- 172.21.15.32:0/1775128218 wait complete. 2024-03-16T19:56:17.926 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-16T19:56:17.939 INFO:tasks.ceph.mds.f.smithi059.stdout:starting mds.f at 2024-03-16T19:56:18.064 INFO:tasks.ceph.mds.b.smithi059.stdout:starting mds.b at 2024-03-16T19:56:19.092 INFO:tasks.ceph.mds.e.smithi032.stdout:starting mds.e at 2024-03-16T19:56:19.125 INFO:tasks.ceph.mds.d.smithi059.stdout:starting mds.d at 2024-03-16T19:56:19.179 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-16T19:56:19.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.496+0000 7f20df2ba640 1 Processor -- start 2024-03-16T19:56:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.496+0000 7f20df2ba640 1 -- start start 2024-03-16T19:56:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20df2ba640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20d8154350 0x7f20d806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20df2ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d80c2d70 0x7f20d815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20df2ba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d806bae0 con 0x7f20d8072a10 2024-03-16T19:56:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20df2ba640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d815a540 con 0x7f20d8154350 2024-03-16T19:56:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20df2ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d80c7fd0 con 0x7f20d80c2d70 2024-03-16T19:56:19.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20dd056640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20d8154350 0x7f20d806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20dd056640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20d8154350 0x7f20d806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44056/0 (socket says 172.21.15.32:44056) 2024-03-16T19:56:19.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20dd056640 1 -- 172.21.15.32:0/3517011398 learned_addr learned my addr 172.21.15.32:0/3517011398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:19.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.500+0000 7f20dc855640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d80c2d70 0x7f20d815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.504+0000 7f20dd056640 1 -- 172.21.15.32:0/3517011398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d80c2d70 msgr2=0x7f20d815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.504+0000 7f20dd056640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d80c2d70 0x7f20d815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.504+0000 7f20dd056640 1 -- 172.21.15.32:0/3517011398 >> v1:172.21.15.32:6789/0 conn(0x7f20d8072a10 legacy=0x7f20d80c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.504+0000 7f20dd056640 1 -- 172.21.15.32:0/3517011398 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d806bd10 con 0x7f20d8154350 2024-03-16T19:56:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.508+0000 7f20dd056640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8154350 0x7f20d806c250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009870 tx=0x7f20d001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b92483313490de server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20d7fff640 1 -- 172.21.15.32:0/3517011398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00ab600 con 0x7f20d8154350 2024-03-16T19:56:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20d7fff640 1 -- 172.21.15.32:0/3517011398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00a7050 con 0x7f20d8154350 2024-03-16T19:56:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20d7fff640 1 -- 172.21.15.32:0/3517011398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00ab980 con 0x7f20d8154350 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 -- 172.21.15.32:0/3517011398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8154350 msgr2=0x7f20d806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8154350 0x7f20d806c250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009870 tx=0x7f20d001c4d0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 -- 172.21.15.32:0/3517011398 shutdown_connections 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d80c2d70 0x7f20d815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 --2- 172.21.15.32:0/3517011398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8154350 0x7f20d806c250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 -- 172.21.15.32:0/3517011398 >> 172.21.15.32:0/3517011398 conn(0x7f20d806afc0 msgr2=0x7f20d806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 -- 172.21.15.32:0/3517011398 shutdown_connections 2024-03-16T19:56:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.512+0000 7f20df2ba640 1 -- 172.21.15.32:0/3517011398 wait complete. 2024-03-16T19:56:19.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 Processor -- start 2024-03-16T19:56:19.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 -- start start 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d80c2d70 0x7f20d8155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d8154350 0x7f20d81477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d81555b0 con 0x7f20d8072a10 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d815c310 con 0x7f20d80c2d70 2024-03-16T19:56:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20df2ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d815ce90 con 0x7f20d8154350 2024-03-16T19:56:19.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20dd056640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20dd056640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44072/0 (socket says 172.21.15.32:44072) 2024-03-16T19:56:19.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20dd056640 1 -- 172.21.15.32:0/462384180 learned_addr learned my addr 172.21.15.32:0/462384180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:19.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20dd857640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d8154350 0x7f20d81477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.516+0000 7f20dc855640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d80c2d70 0x7f20d8155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.520+0000 7f20dd056640 1 -- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d8154350 msgr2=0x7f20d81477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.520+0000 7f20dd056640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d8154350 0x7f20d81477b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.520+0000 7f20dd056640 1 -- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d80c2d70 msgr2=0x7f20d8155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.520+0000 7f20dd056640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d80c2d70 0x7f20d8155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.520+0000 7f20dd056640 1 -- 172.21.15.32:0/462384180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00094d0 con 0x7f20d8072a10 2024-03-16T19:56:19.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.524+0000 7f20dd056640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20d001a100 tx=0x7f20d0019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:19.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00b5400 con 0x7f20d8072a10 2024-03-16T19:56:19.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00b5560 con 0x7f20d8072a10 2024-03-16T19:56:19.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d80b1f40 con 0x7f20d8072a10 2024-03-16T19:56:19.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00b5910 con 0x7f20d8072a10 2024-03-16T19:56:19.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d815c840 con 0x7f20d8072a10 2024-03-16T19:56:19.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.528+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d80674e0 con 0x7f20d8072a10 2024-03-16T19:56:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.844+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f20d00b5a70 con 0x7f20d8072a10 2024-03-16T19:56:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.844+0000 7f20d5ffb640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 0x7f20c4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.844+0000 7f20dc855640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 0x7f20c4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.844+0000 7f20dc855640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 0x7f20c4069380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20d80c7a70 tx=0x7f20cc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.884+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20d00f0260 con 0x7f20d8072a10 2024-03-16T19:56:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:19.888+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f20d00ec500 con 0x7f20d8072a10 2024-03-16T19:56:20.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.100+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20d80c7c10 con 0x7f20d8072a10 2024-03-16T19:56:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.112+0000 7f20d5ffb640 1 -- 172.21.15.32:0/462384180 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f20d00b0020 con 0x7f20d8072a10 2024-03-16T19:56:20.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:20.084 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T19:56:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 msgr2=0x7f20c4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 0x7f20c4069380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20d80c7a70 tx=0x7f20cc009190 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 msgr2=0x7f20d8153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20d001a100 tx=0x7f20d0019270 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 shutdown_connections 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20c4066ec0 0x7f20c4069380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d8154350 0x7f20d81477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d80c2d70 0x7f20d8155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 --2- 172.21.15.32:0/462384180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d8072a10 0x7f20d8153740 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 >> 172.21.15.32:0/462384180 conn(0x7f20d806afc0 msgr2=0x7f20d80c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 shutdown_connections 2024-03-16T19:56:20.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:20.116+0000 7f20df2ba640 1 -- 172.21.15.32:0/462384180 wait complete. 2024-03-16T19:56:22.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:22.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.272+0000 7f740f574640 1 Processor -- start 2024-03-16T19:56:22.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.272+0000 7f740f574640 1 -- start start 2024-03-16T19:56:22.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740f574640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7408150a60 0x7f74080c5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740f574640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74081565b0 0x7f7408156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740f574640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080c74b0 con 0x7f74080c5780 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740f574640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74080c3bf0 con 0x7f74081565b0 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740f574640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74080729a0 con 0x7f7408150a60 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740db11640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74081565b0 0x7f7408156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740db11640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74081565b0 0x7f7408156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44078/0 (socket says 172.21.15.32:44078) 2024-03-16T19:56:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.276+0000 7f740db11640 1 -- 172.21.15.32:0/2292139554 learned_addr learned my addr 172.21.15.32:0/2292139554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.280+0000 7f740db11640 1 -- 172.21.15.32:0/2292139554 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7408150a60 msgr2=0x7f74080c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.280+0000 7f740db11640 1 --2- 172.21.15.32:0/2292139554 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7408150a60 0x7f74080c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.280+0000 7f740db11640 1 -- 172.21.15.32:0/2292139554 >> v1:172.21.15.59:6789/0 conn(0x7f74080c5780 legacy=0x7f74080c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.280+0000 7f740db11640 1 -- 172.21.15.32:0/2292139554 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408072c80 con 0x7f74081565b0 2024-03-16T19:56:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.284+0000 7f740db11640 1 --2- 172.21.15.32:0/2292139554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f7408156a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f740000af60 tx=0x7f7400020330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0131fe72f00d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:22.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f73fffff640 1 -- 172.21.15.32:0/2292139554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740000a2d0 con 0x7f74081565b0 2024-03-16T19:56:22.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f73fffff640 1 -- 172.21.15.32:0/2292139554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000ac050 con 0x7f74081565b0 2024-03-16T19:56:22.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f73fffff640 1 -- 172.21.15.32:0/2292139554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400008720 con 0x7f74081565b0 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- 172.21.15.32:0/2292139554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 msgr2=0x7f7408156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 --2- 172.21.15.32:0/2292139554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f7408156a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f740000af60 tx=0x7f7400020330 comp rx=0 tx=0).stop 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- 172.21.15.32:0/2292139554 shutdown_connections 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 --2- 172.21.15.32:0/2292139554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f7408156a20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 --2- 172.21.15.32:0/2292139554 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7408150a60 0x7f74080c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- 172.21.15.32:0/2292139554 >> 172.21.15.32:0/2292139554 conn(0x7f74080cb470 msgr2=0x7f74080cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- 172.21.15.32:0/2292139554 shutdown_connections 2024-03-16T19:56:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- 172.21.15.32:0/2292139554 wait complete. 2024-03-16T19:56:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 Processor -- start 2024-03-16T19:56:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.288+0000 7f740f574640 1 -- start start 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74080c5780 0x7f740814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7408150a60 0x7f74080c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080cae10 con 0x7f74081565b0 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740814eb60 con 0x7f7408150a60 2024-03-16T19:56:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740f574640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74081df450 con 0x7f74080c5780 2024-03-16T19:56:22.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740db11640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:22.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740db11640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44084/0 (socket says 172.21.15.32:44084) 2024-03-16T19:56:22.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740db11640 1 -- 172.21.15.32:0/1894230764 learned_addr learned my addr 172.21.15.32:0/1894230764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:22.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740cb0f640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7408150a60 0x7f74080c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:22.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.292+0000 7f740d310640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74080c5780 0x7f740814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 -- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74080c5780 msgr2=0x7f740814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74080c5780 0x7f740814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 -- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7408150a60 msgr2=0x7f74080c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7408150a60 0x7f74080c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 -- 172.21.15.32:0/1894230764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740000ac40 con 0x7f74081565b0 2024-03-16T19:56:22.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.296+0000 7f740db11640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f740001df60 tx=0x7f740001d0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400008b80 con 0x7f74081565b0 2024-03-16T19:56:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7400008ce0 con 0x7f74081565b0 2024-03-16T19:56:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000bb690 con 0x7f74081565b0 2024-03-16T19:56:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7408143aa0 con 0x7f74081565b0 2024-03-16T19:56:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74080c93a0 con 0x7f74081565b0 2024-03-16T19:56:22.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.304+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74080c95a0 con 0x7f74081565b0 2024-03-16T19:56:22.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.324+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f74000b7070 con 0x7f74081565b0 2024-03-16T19:56:22.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.328+0000 7f73fdffb640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 0x7f73ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:22.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.328+0000 7f740d310640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 0x7f73ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:22.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.328+0000 7f740d310640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 0x7f73ec069300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.664+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74000f5150 con 0x7f74081565b0 2024-03-16T19:56:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.664+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f74000f0cf0 con 0x7f74081565b0 2024-03-16T19:56:22.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.880+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74080c7030 con 0x7f74081565b0 2024-03-16T19:56:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f73fdffb640 1 -- 172.21.15.32:0/1894230764 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f74000b5020 con 0x7f74081565b0 2024-03-16T19:56:22.867 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:22.867 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T19:56:22.868 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 msgr2=0x7f73ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 0x7f73ec069300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404008040 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 msgr2=0x7f74080c9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f740001df60 tx=0x7f740001d0d0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 shutdown_connections 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f73ec066e40 0x7f73ec069300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74081565b0 0x7f74080c9dc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7408150a60 0x7f74080c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 --2- 172.21.15.32:0/1894230764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74080c5780 0x7f740814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 >> 172.21.15.32:0/1894230764 conn(0x7f74080cb470 msgr2=0x7f74080c61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 shutdown_connections 2024-03-16T19:56:22.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:22.896+0000 7f740f574640 1 -- 172.21.15.32:0/1894230764 wait complete. 2024-03-16T19:56:24.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 Processor -- start 2024-03-16T19:56:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 -- start start 2024-03-16T19:56:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35f00714f0 0x7f35f00ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 0x7f35f00cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00ca1d0 con 0x7f35f0071110 2024-03-16T19:56:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00c34a0 con 0x7f35f00714f0 2024-03-16T19:56:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f66bc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c9ba0 con 0x7f35f00c2910 2024-03-16T19:56:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35effff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35f0071110 0x7f35f00c82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60952/0 (socket says 172.21.15.32:60952) 2024-03-16T19:56:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35effff640 1 -- 172.21.15.32:0/3778179256 learned_addr learned my addr 172.21.15.32:0/3778179256 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35ef7fe640 1 --2- 172.21.15.32:0/3778179256 >> v2:172.21.15.59:3300/0 conn(0x7f35f00714f0 0x7f35f00ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.052+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 0x7f35f00cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.056+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325862662 0 0) 0x7f35f00ca1d0 con 0x7f35f0071110 2024-03-16T19:56:25.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.056+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc003530 con 0x7f35f0071110 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2864575272 0 0) 0x7f35dc003530 con 0x7f35f0071110 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 msgr2=0x7f35f00cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 0x7f35f00cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 >> v2:172.21.15.59:3300/0 conn(0x7f35f00714f0 msgr2=0x7f35f00ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 --2- 172.21.15.32:0/3778179256 >> v2:172.21.15.59:3300/0 conn(0x7f35f00714f0 0x7f35f00ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00c2ee0 con 0x7f35f0071110 2024-03-16T19:56:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f35e8001a00 con 0x7f35f0071110 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 >> v1:172.21.15.32:6789/0 conn(0x7f35f0071110 legacy=0x7f35f00c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35dc003980 0x7f35dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 0x7f35dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35dc004d10 0x7f35dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c34a0 con 0x7f35dc004d10 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c9ba0 con 0x7f35dc003980 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc00b990 con 0x7f35f00c2910 2024-03-16T19:56:25.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.060+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35dc004d10 0x7f35dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.064+0000 7f35f4c59640 1 -- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 msgr2=0x7f35dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:25.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.064+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35f00c2910 0x7f35dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.064+0000 7f35f4c59640 1 -- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35dc003980 msgr2=0x7f35dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:25.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.064+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35dc003980 0x7f35dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.064+0000 7f35f4c59640 1 -- 172.21.15.32:0/3778179256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00c2ee0 con 0x7f35dc004d10 2024-03-16T19:56:25.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.068+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35dc004d10 0x7f35dc005190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35e001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36a57d6a7b2dc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e00a56a0 con 0x7f35dc004d10 2024-03-16T19:56:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e00a5800 con 0x7f35dc004d10 2024-03-16T19:56:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35eeffd640 1 -- 172.21.15.32:0/3778179256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e00a5bf0 con 0x7f35dc004d10 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 -- 172.21.15.32:0/3778179256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35dc004d10 msgr2=0x7f35dc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3778179256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35dc004d10 0x7f35dc005190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35e001cab0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f4c59640 1 -- 172.21.15.32:0/3778179256 reap_dead start 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 -- 172.21.15.32:0/3778179256 shutdown_connections 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 -- 172.21.15.32:0/3778179256 >> 172.21.15.32:0/3778179256 conn(0x7f35f006afc0 msgr2=0x7f35f006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 -- 172.21.15.32:0/3778179256 shutdown_connections 2024-03-16T19:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.072+0000 7f35f66bc640 1 -- 172.21.15.32:0/3778179256 wait complete. 2024-03-16T19:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 Processor -- start 2024-03-16T19:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 -- start start 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35dc004d10 0x7f35f00ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35f0156ba0 0x7f35f01f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00cdff0 con 0x7f35f00cb990 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00cc8f0 con 0x7f35f0156ba0 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f66bc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01f0c20 con 0x7f35dc004d10 2024-03-16T19:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f4c59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f4c59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44104/0 (socket says 172.21.15.32:44104) 2024-03-16T19:56:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35f4c59640 1 -- 172.21.15.32:0/3187762746 learned_addr learned my addr 172.21.15.32:0/3187762746 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35ef7fe640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35f0156ba0 0x7f35f01f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.076+0000 7f35effff640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35dc004d10 0x7f35f00ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.080+0000 7f35f4c59640 1 -- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35dc004d10 msgr2=0x7f35f00ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.080+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35dc004d10 0x7f35f00ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.080+0000 7f35f4c59640 1 -- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35f0156ba0 msgr2=0x7f35f01f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.080+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35f0156ba0 0x7f35f01f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.080+0000 7f35f4c59640 1 -- 172.21.15.32:0/3187762746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00b2e20 con 0x7f35f00cb990 2024-03-16T19:56:25.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.084+0000 7f35f4c59640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35dc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:25.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0004230 con 0x7f35f00cb990 2024-03-16T19:56:25.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0004390 con 0x7f35f00cb990 2024-03-16T19:56:25.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0003820 con 0x7f35f00cb990 2024-03-16T19:56:25.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f014c620 con 0x7f35f00cb990 2024-03-16T19:56:25.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f014c870 con 0x7f35f00cb990 2024-03-16T19:56:25.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.088+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f00ce130 con 0x7f35f00cb990 2024-03-16T19:56:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.380+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f35e0003980 con 0x7f35f00cb990 2024-03-16T19:56:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.380+0000 7f35ecff9640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 0x7f35d0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.380+0000 7f35effff640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 0x7f35d0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.380+0000 7f35effff640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 0x7f35d0069350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35e80013e0 tx=0x7f35e8007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:25.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.408+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f35e00ef750 con 0x7f35f00cb990 2024-03-16T19:56:25.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.408+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f35e00eb9f0 con 0x7f35f00cb990 2024-03-16T19:56:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.628+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35f00ce130 con 0x7f35f00cb990 2024-03-16T19:56:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.896+0000 7f35ecff9640 1 -- 172.21.15.32:0/3187762746 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f35f00ce130 con 0x7f35f00cb990 2024-03-16T19:56:25.868 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:25.868 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4242,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T19:56:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 msgr2=0x7f35d0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 0x7f35d0069350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35e80013e0 tx=0x7f35e8007d50 comp rx=0 tx=0).stop 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 msgr2=0x7f35f01f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35dc005800 comp rx=0 tx=0).stop 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 shutdown_connections 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35d0066e90 0x7f35d0069350 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f00cb990 0x7f35f01f31d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35f0156ba0 0x7f35f01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 --2- 172.21.15.32:0/3187762746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35dc004d10 0x7f35f00ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 >> 172.21.15.32:0/3187762746 conn(0x7f35f006afc0 msgr2=0x7f35f00ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 shutdown_connections 2024-03-16T19:56:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:25.900+0000 7f35f66bc640 1 -- 172.21.15.32:0/3187762746 wait complete. 2024-03-16T19:56:27.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.056+0000 7f2873a79640 1 Processor -- start 2024-03-16T19:56:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2873a79640 1 -- start start 2024-03-16T19:56:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2873a79640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f286c06f4f0 0x7f286c165960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2873a79640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c15af90 con 0x7f286c0c7230 2024-03-16T19:56:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2873a79640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c0c5fa0 con 0x7f286c153fe0 2024-03-16T19:56:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2873a79640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c0c7040 con 0x7f286c06f4f0 2024-03-16T19:56:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2872016640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f286c06f4f0 0x7f286c165960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2872016640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f286c06f4f0 0x7f286c165960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44106/0 (socket says 172.21.15.32:44106) 2024-03-16T19:56:28.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.060+0000 7f2872016640 1 -- 172.21.15.32:0/530722054 learned_addr learned my addr 172.21.15.32:0/530722054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.064+0000 7f2872016640 1 -- 172.21.15.32:0/530722054 >> v1:172.21.15.59:6789/0 conn(0x7f286c153fe0 legacy=0x7f286c071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.064+0000 7f2872016640 1 -- 172.21.15.32:0/530722054 >> v1:172.21.15.32:6789/0 conn(0x7f286c0c7230 legacy=0x7f286c06e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.064+0000 7f2872016640 1 -- 172.21.15.32:0/530722054 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c0c6280 con 0x7f286c06f4f0 2024-03-16T19:56:28.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.068+0000 7f2872016640 1 --2- 172.21.15.32:0/530722054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c06f4f0 0x7f286c165960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e50 tx=0x7f286401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83ec26d1fa5aa75 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2870813640 1 -- 172.21.15.32:0/530722054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28640a58d0 con 0x7f286c06f4f0 2024-03-16T19:56:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2870813640 1 -- 172.21.15.32:0/530722054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28640a5a30 con 0x7f286c06f4f0 2024-03-16T19:56:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2870813640 1 -- 172.21.15.32:0/530722054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28640a5dc0 con 0x7f286c06f4f0 2024-03-16T19:56:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 -- 172.21.15.32:0/530722054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c06f4f0 msgr2=0x7f286c165960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 --2- 172.21.15.32:0/530722054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c06f4f0 0x7f286c165960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e50 tx=0x7f286401ca00 comp rx=0 tx=0).stop 2024-03-16T19:56:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 -- 172.21.15.32:0/530722054 shutdown_connections 2024-03-16T19:56:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 --2- 172.21.15.32:0/530722054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c06f4f0 0x7f286c165960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 -- 172.21.15.32:0/530722054 >> 172.21.15.32:0/530722054 conn(0x7f286c0cb470 msgr2=0x7f286c0cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:28.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 -- 172.21.15.32:0/530722054 shutdown_connections 2024-03-16T19:56:28.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.076+0000 7f2873a79640 1 -- 172.21.15.32:0/530722054 wait complete. 2024-03-16T19:56:28.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 Processor -- start 2024-03-16T19:56:28.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 -- start start 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f286c06f4f0 0x7f286c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f286c153fe0 0x7f286c140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c146660 con 0x7f286c0c7230 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c145790 con 0x7f286c153fe0 2024-03-16T19:56:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2873a79640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1406c0 con 0x7f286c06f4f0 2024-03-16T19:56:28.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2871014640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2871014640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44108/0 (socket says 172.21.15.32:44108) 2024-03-16T19:56:28.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2871014640 1 -- 172.21.15.32:0/1831963549 learned_addr learned my addr 172.21.15.32:0/1831963549 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:28.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.080+0000 7f2871815640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f286c06f4f0 0x7f286c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2872016640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f286c153fe0 0x7f286c140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2871014640 1 -- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f286c06f4f0 msgr2=0x7f286c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2871014640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f286c06f4f0 0x7f286c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2871014640 1 -- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f286c153fe0 msgr2=0x7f286c140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2871014640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f286c153fe0 0x7f286c140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.084+0000 7f2871014640 1 -- 172.21.15.32:0/1831963549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864009b30 con 0x7f286c0c7230 2024-03-16T19:56:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.088+0000 7f2871014640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c500 tx=0x7f285c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c005220 con 0x7f286c0c7230 2024-03-16T19:56:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c005380 con 0x7f286c0c7230 2024-03-16T19:56:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c007270 con 0x7f286c0c7230 2024-03-16T19:56:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c1460a0 con 0x7f286c0c7230 2024-03-16T19:56:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c145140 con 0x7f286c0c7230 2024-03-16T19:56:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.092+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c06a340 con 0x7f286c0c7230 2024-03-16T19:56:28.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.400+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f285c0073d0 con 0x7f286c0c7230 2024-03-16T19:56:28.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.400+0000 7f28627fc640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 0x7f2850069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.400+0000 7f2871815640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 0x7f2850069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.400+0000 7f2871815640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 0x7f2850069300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2868007fd0 tx=0x7f2868009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:28.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.432+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f285c007680 con 0x7f286c0c7230 2024-03-16T19:56:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.432+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f285c051eb0 con 0x7f286c0c7230 2024-03-16T19:56:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.656+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286c153810 con 0x7f286c0c7230 2024-03-16T19:56:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f28627fc640 1 -- 172.21.15.32:0/1831963549 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f285c055570 con 0x7f286c0c7230 2024-03-16T19:56:28.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:28.679 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-16T19:56:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-03-16T19:56:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 msgr2=0x7f2850069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 0x7f2850069300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2868007fd0 tx=0x7f2868009040 comp rx=0 tx=0).stop 2024-03-16T19:56:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 msgr2=0x7f286c1459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c500 tx=0x7f285c00c9c0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 shutdown_connections 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2850066e40 0x7f2850069300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f286c153fe0 0x7f286c140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f286c0c7230 0x7f286c1459d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 --2- 172.21.15.32:0/1831963549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f286c06f4f0 0x7f286c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 >> 172.21.15.32:0/1831963549 conn(0x7f286c0cb470 msgr2=0x7f286c06e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 shutdown_connections 2024-03-16T19:56:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.708+0000 7f2873a79640 1 -- 172.21.15.32:0/1831963549 wait complete. 2024-03-16T19:56:28.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:28.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 Processor -- start 2024-03-16T19:56:28.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 -- start start 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc978071510 0x7fc9780c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9780c8fe0 0x7fc9780c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780c2940 con 0x7fc978071130 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9780c8a30 con 0x7fc978071510 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97cf80640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9780c9950 con 0x7fc9780c8fe0 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97659c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc978071130 0x7fc9780c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60974/0 (socket says 172.21.15.32:60974) 2024-03-16T19:56:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.856+0000 7fc97659c640 1 -- 172.21.15.32:0/3120726654 learned_addr learned my addr 172.21.15.32:0/3120726654 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc96dd9b640 1 --2- 172.21.15.32:0/3120726654 >> v2:172.21.15.32:3300/0 conn(0x7fc978071510 0x7fc9780c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc975d9b640 1 -- 172.21.15.32:0/3120726654 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890194292 0 0) 0x7fc9780c2940 con 0x7fc978071130 2024-03-16T19:56:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc975d9b640 1 -- 172.21.15.32:0/3120726654 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964003530 con 0x7fc978071130 2024-03-16T19:56:28.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc96dd9b640 1 -- 172.21.15.32:0/3120726654 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9780c8fe0 msgr2=0x7fc9780c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:28.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc96dd9b640 1 --2- 172.21.15.32:0/3120726654 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9780c8fe0 0x7fc9780c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc96dd9b640 1 -- 172.21.15.32:0/3120726654 >> v1:172.21.15.32:6789/0 conn(0x7fc978071130 legacy=0x7fc9780c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.860+0000 7fc96dd9b640 1 -- 172.21.15.32:0/3120726654 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780b2de0 con 0x7fc978071510 2024-03-16T19:56:28.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.868+0000 7fc96dd9b640 1 --2- 172.21.15.32:0/3120726654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978071510 0x7fc9780c9b90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9600097c0 tx=0x7fc96001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93302a4031b6a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:28.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc975d9b640 1 -- 172.21.15.32:0/3120726654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9600a8990 con 0x7fc978071510 2024-03-16T19:56:28.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc975d9b640 1 -- 172.21.15.32:0/3120726654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9600a8af0 con 0x7fc978071510 2024-03-16T19:56:28.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc975d9b640 1 -- 172.21.15.32:0/3120726654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9600a8e80 con 0x7fc978071510 2024-03-16T19:56:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 -- 172.21.15.32:0/3120726654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978071510 msgr2=0x7fc9780c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 --2- 172.21.15.32:0/3120726654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978071510 0x7fc9780c9b90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9600097c0 tx=0x7fc96001ec90 comp rx=0 tx=0).stop 2024-03-16T19:56:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 -- 172.21.15.32:0/3120726654 shutdown_connections 2024-03-16T19:56:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 --2- 172.21.15.32:0/3120726654 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9780c8fe0 0x7fc9780c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 --2- 172.21.15.32:0/3120726654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978071510 0x7fc9780c9b90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 -- 172.21.15.32:0/3120726654 >> 172.21.15.32:0/3120726654 conn(0x7fc97806b190 msgr2=0x7fc97806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 -- 172.21.15.32:0/3120726654 shutdown_connections 2024-03-16T19:56:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.872+0000 7fc97cf80640 1 -- 172.21.15.32:0/3120726654 wait complete. 2024-03-16T19:56:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 Processor -- start 2024-03-16T19:56:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 -- start start 2024-03-16T19:56:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc978071130 0x7fc9781ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc978071510 0x7fc9781f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97815a900 con 0x7fc9780c8fe0 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97815b160 con 0x7fc978071510 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc97cf80640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97815ba70 con 0x7fc978071130 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc976d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc976d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44126/0 (socket says 172.21.15.32:44126) 2024-03-16T19:56:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.876+0000 7fc976d9d640 1 -- 172.21.15.32:0/1313147627 learned_addr learned my addr 172.21.15.32:0/1313147627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.880+0000 7fc976d9d640 1 -- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc978071130 msgr2=0x7fc9781ee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.880+0000 7fc976d9d640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc978071130 0x7fc9781ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.880+0000 7fc976d9d640 1 -- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc978071510 msgr2=0x7fc9781f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.880+0000 7fc976d9d640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc978071510 0x7fc9781f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.880+0000 7fc976d9d640 1 -- 172.21.15.32:0/1313147627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc960009420 con 0x7fc9780c8fe0 2024-03-16T19:56:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.884+0000 7fc976d9d640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc97000b9a0 tx=0x7fc97000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97000cba0 con 0x7fc9780c8fe0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97000cd20 con 0x7fc9780c8fe0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970014680 con 0x7fc9780c8fe0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97815bcc0 con 0x7fc9780c8fe0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97815ab10 con 0x7fc9780c8fe0 2024-03-16T19:56:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:28.888+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc978068000 con 0x7fc9780c8fe0 2024-03-16T19:56:29.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.180+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc97001d3b0 con 0x7fc9780c8fe0 2024-03-16T19:56:29.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.180+0000 7fc96f7fe640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 0x7fc958069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.180+0000 7fc97659c640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 0x7fc958069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.184+0000 7fc97659c640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 0x7fc958069380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc968007fd0 tx=0x7fc968007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:29.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.224+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc970056070 con 0x7fc9780c8fe0 2024-03-16T19:56:29.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.228+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc970051c10 con 0x7fc9780c8fe0 2024-03-16T19:56:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.404+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc978151b30 con 0x7fc9780c8fe0 2024-03-16T19:56:29.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.468+0000 7fc96f7fe640 1 -- 172.21.15.32:0/1313147627 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12824 (secure 0 0 0) 0x7fc970019020 con 0x7fc9780c8fe0 2024-03-16T19:56:29.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:29.441 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:55:56.041430+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 msgr2=0x7fc958069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 0x7fc958069380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc968007fd0 tx=0x7fc968007900 comp rx=0 tx=0).stop 2024-03-16T19:56:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 msgr2=0x7fc9781f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc97000b9a0 tx=0x7fc97000be60 comp rx=0 tx=0).stop 2024-03-16T19:56:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 shutdown_connections 2024-03-16T19:56:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc958066ec0 0x7fc958069380 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9780c8fe0 0x7fc9781f0f50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc978071510 0x7fc9781f0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 --2- 172.21.15.32:0/1313147627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc978071130 0x7fc9781ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 >> 172.21.15.32:0/1313147627 conn(0x7fc97806b190 msgr2=0x7fc97814b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 shutdown_connections 2024-03-16T19:56:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.472+0000 7fc97cf80640 1 -- 172.21.15.32:0/1313147627 wait complete. 2024-03-16T19:56:29.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:56:29.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:56:29.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 Processor -- start 2024-03-16T19:56:29.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 -- start start 2024-03-16T19:56:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0234150e20 0x7f0234073060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02340c7590 0x7f02340c2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02340735a0 con 0x7f02340c2eb0 2024-03-16T19:56:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340c32c0 con 0x7f0234150e20 2024-03-16T19:56:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f023973e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0234156bb0 con 0x7f02340c7590 2024-03-16T19:56:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.624+0000 7f02327fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02340c7590 0x7f02340c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.628+0000 7f02337fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02340c2eb0 0x7f02341572a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38060/0 (socket says 172.21.15.32:38060) 2024-03-16T19:56:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.628+0000 7f02337fe640 1 -- 172.21.15.32:0/3198936993 learned_addr learned my addr 172.21.15.32:0/3198936993 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:29.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.628+0000 7f0232ffd640 1 --2- 172.21.15.32:0/3198936993 >> v2:172.21.15.32:3300/0 conn(0x7f0234150e20 0x7f0234073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.632+0000 7f0232ffd640 1 -- 172.21.15.32:0/3198936993 >> v2:172.21.15.59:3300/0 conn(0x7f02340c7590 msgr2=0x7f02340c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:29.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.632+0000 7f0232ffd640 1 --2- 172.21.15.32:0/3198936993 >> v2:172.21.15.59:3300/0 conn(0x7f02340c7590 0x7f02340c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.632+0000 7f0232ffd640 1 -- 172.21.15.32:0/3198936993 >> v1:172.21.15.32:6789/0 conn(0x7f02340c2eb0 legacy=0x7f02341572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.632+0000 7f0232ffd640 1 -- 172.21.15.32:0/3198936993 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234156dc0 con 0x7f0234150e20 2024-03-16T19:56:29.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.636+0000 7f0232ffd640 1 --2- 172.21.15.32:0/3198936993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0234150e20 0x7f0234073060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02340c9320 tx=0x7f022401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf5391c711e314 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f0231ffb640 1 -- 172.21.15.32:0/3198936993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240ab600 con 0x7f0234150e20 2024-03-16T19:56:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f0231ffb640 1 -- 172.21.15.32:0/3198936993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02240a7050 con 0x7f0234150e20 2024-03-16T19:56:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f0231ffb640 1 -- 172.21.15.32:0/3198936993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240ab980 con 0x7f0234150e20 2024-03-16T19:56:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- 172.21.15.32:0/3198936993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0234150e20 msgr2=0x7f0234073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- 172.21.15.32:0/3198936993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0234150e20 0x7f0234073060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02340c9320 tx=0x7f022401c480 comp rx=0 tx=0).stop 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- 172.21.15.32:0/3198936993 shutdown_connections 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- 172.21.15.32:0/3198936993 >> v2:172.21.15.59:3300/0 conn(0x7f02340c7590 0x7f02340c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- 172.21.15.32:0/3198936993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0234150e20 0x7f0234073060 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- 172.21.15.32:0/3198936993 >> 172.21.15.32:0/3198936993 conn(0x7f023406afc0 msgr2=0x7f023406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- 172.21.15.32:0/3198936993 shutdown_connections 2024-03-16T19:56:29.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- 172.21.15.32:0/3198936993 wait complete. 2024-03-16T19:56:29.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 Processor -- start 2024-03-16T19:56:29.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- start start 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02340c7590 0x7f02341f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0234150e20 0x7f02341f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f6bc0 con 0x7f02340c2eb0 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f2840 con 0x7f02340c7590 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f023973e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f6620 con 0x7f0234150e20 2024-03-16T19:56:29.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.644+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32942/0 (socket says 172.21.15.32:32942) 2024-03-16T19:56:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 -- 172.21.15.32:0/4221345567 learned_addr learned my addr 172.21.15.32:0/4221345567 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f02337fe640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0234150e20 0x7f02341f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f02327fc640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02340c7590 0x7f02341f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 -- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0234150e20 msgr2=0x7f02341f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0234150e20 0x7f02341f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 -- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02340c7590 msgr2=0x7f02341f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:29.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02340c7590 0x7f02341f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:29.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.648+0000 7f0232ffd640 1 -- 172.21.15.32:0/4221345567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0224009420 con 0x7f02340c2eb0 2024-03-16T19:56:29.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.652+0000 7f0232ffd640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02240ae040 tx=0x7f0224019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240b4410 con 0x7f02340c2eb0 2024-03-16T19:56:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02240b4570 con 0x7f02340c2eb0 2024-03-16T19:56:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240b4900 con 0x7f02340c2eb0 2024-03-16T19:56:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02341f68e0 con 0x7f02340c2eb0 2024-03-16T19:56:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02341501a0 con 0x7f02340c2eb0 2024-03-16T19:56:29.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.656+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02340c3970 con 0x7f02340c2eb0 2024-03-16T19:56:29.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.676+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02240b2040 con 0x7f02340c2eb0 2024-03-16T19:56:29.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.680+0000 7f021b7fe640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 0x7f0220069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.680+0000 7f02327fc640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 0x7f0220069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.680+0000 7f02327fc640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 0x7f0220069350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0234160340 tx=0x7f0228009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:29.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.976+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02240efef0 con 0x7f02340c2eb0 2024-03-16T19:56:29.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:29.976+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f02240ec190 con 0x7f02340c2eb0 2024-03-16T19:56:30.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.156+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0234071940 con 0x7f02340c2eb0 2024-03-16T19:56:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.164+0000 7f021b7fe640 1 -- 172.21.15.32:0/4221345567 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02240f2020 con 0x7f02340c2eb0 2024-03-16T19:56:30.138 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 msgr2=0x7f0220069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 0x7f0220069350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0234160340 tx=0x7f0228009190 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 msgr2=0x7f023414c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02240ae040 tx=0x7f0224019220 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 shutdown_connections 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0220066e90 0x7f0220069350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0234150e20 0x7f02341f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02340c7590 0x7f02341f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 --2- 172.21.15.32:0/4221345567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02340c2eb0 0x7f023414c800 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 >> 172.21.15.32:0/4221345567 conn(0x7f023406afc0 msgr2=0x7f0234160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 shutdown_connections 2024-03-16T19:56:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.168+0000 7f023973e640 1 -- 172.21.15.32:0/4221345567 wait complete. 2024-03-16T19:56:30.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T19:56:30.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 Processor -- start 2024-03-16T19:56:30.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 -- start start 2024-03-16T19:56:30.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7884150c40 0x7f7884073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f788415a420 0x7f788415a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7884151020 con 0x7f78840725e0 2024-03-16T19:56:30.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788415a000 con 0x7f7884150c40 2024-03-16T19:56:30.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.312+0000 7f78889cb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788415add0 con 0x7f788415a420 2024-03-16T19:56:30.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.376+0000 7f788259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7884150c40 0x7f7884073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:30.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.376+0000 7f788259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7884150c40 0x7f7884073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32954/0 (socket says 172.21.15.32:32954) 2024-03-16T19:56:30.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.376+0000 7f788259c640 1 -- 172.21.15.32:0/260858562 learned_addr learned my addr 172.21.15.32:0/260858562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.384+0000 7f788259c640 1 -- 172.21.15.32:0/260858562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f788415a420 msgr2=0x7f788415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:30.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.384+0000 7f788259c640 1 --2- 172.21.15.32:0/260858562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f788415a420 0x7f788415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.384+0000 7f788259c640 1 -- 172.21.15.32:0/260858562 >> v1:172.21.15.59:6789/0 conn(0x7f78840725e0 legacy=0x7f7884070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:30.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.384+0000 7f788259c640 1 -- 172.21.15.32:0/260858562 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884072da0 con 0x7f7884150c40 2024-03-16T19:56:30.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.388+0000 7f788259c640 1 --2- 172.21.15.32:0/260858562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7884150c40 0x7f7884073070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f788406cf60 tx=0x7f787401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3011d60b05d93c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:30.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.396+0000 7f788159a640 1 -- 172.21.15.32:0/260858562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740aa620 con 0x7f7884150c40 2024-03-16T19:56:30.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.396+0000 7f788159a640 1 -- 172.21.15.32:0/260858562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78740aa780 con 0x7f7884150c40 2024-03-16T19:56:30.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.396+0000 7f788159a640 1 -- 172.21.15.32:0/260858562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740aab10 con 0x7f7884150c40 2024-03-16T19:56:30.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- 172.21.15.32:0/260858562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7884150c40 msgr2=0x7f7884073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 --2- 172.21.15.32:0/260858562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7884150c40 0x7f7884073070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f788406cf60 tx=0x7f787401c500 comp rx=0 tx=0).stop 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- 172.21.15.32:0/260858562 shutdown_connections 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 --2- 172.21.15.32:0/260858562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f788415a420 0x7f788415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 --2- 172.21.15.32:0/260858562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7884150c40 0x7f7884073070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- 172.21.15.32:0/260858562 >> 172.21.15.32:0/260858562 conn(0x7f788406c0d0 msgr2=0x7f788406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- 172.21.15.32:0/260858562 shutdown_connections 2024-03-16T19:56:30.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- 172.21.15.32:0/260858562 wait complete. 2024-03-16T19:56:30.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 Processor -- start 2024-03-16T19:56:30.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.400+0000 7f78889cb640 1 -- start start 2024-03-16T19:56:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 0x7f788415cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7884150c40 0x7f78840c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 0x7f78840c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78840cb1e0 con 0x7f78840725e0 2024-03-16T19:56:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78840cbbc0 con 0x7f788415a420 2024-03-16T19:56:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f78889cb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78840cc0d0 con 0x7f7884150c40 2024-03-16T19:56:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f7882d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 0x7f78840c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f788259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 0x7f788415cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f7882d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 0x7f78840c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43614/0 (socket says 172.21.15.32:43614) 2024-03-16T19:56:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.404+0000 7f7882d9d640 1 -- 172.21.15.32:0/544840491 learned_addr learned my addr 172.21.15.32:0/544840491 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.408+0000 7f788259c640 1 -- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7884150c40 msgr2=0x7f78840c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.408+0000 7f788259c640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7884150c40 0x7f78840c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.408+0000 7f788259c640 1 -- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 msgr2=0x7f78840c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.408+0000 7f788259c640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 0x7f78840c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.408+0000 7f788259c640 1 -- 172.21.15.32:0/544840491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78740094d0 con 0x7f78840725e0 2024-03-16T19:56:30.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.412+0000 7f788259c640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 0x7f788415cd40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78740ab000 tx=0x7f787401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740b3800 con 0x7f78840725e0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78740b3960 con 0x7f78840725e0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740b3d10 con 0x7f78840725e0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78840cc300 con 0x7f78840725e0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78840ca840 con 0x7f78840725e0 2024-03-16T19:56:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.420+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7884145470 con 0x7f78840725e0 2024-03-16T19:56:30.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.704+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f78740c7420 con 0x7f78840725e0 2024-03-16T19:56:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.704+0000 7f7872ffd640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 0x7f7864069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:30.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.704+0000 7f7881d9b640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 0x7f7864069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:30.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.708+0000 7f7881d9b640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 0x7f7864069350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7884144400 tx=0x7f7878006bb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.728+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78740eff00 con 0x7f78840725e0 2024-03-16T19:56:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.728+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f78740ec1a0 con 0x7f78840725e0 2024-03-16T19:56:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.916+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7884068000 con 0x7f78840725e0 2024-03-16T19:56:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.924+0000 7f7872ffd640 1 -- 172.21.15.32:0/544840491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f78740ef860 con 0x7f78840725e0 2024-03-16T19:56:30.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-16T19:56:30.897 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 msgr2=0x7f7864069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 0x7f7864069350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7884144400 tx=0x7f7878006bb0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 msgr2=0x7f788415cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 0x7f788415cd40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78740ab000 tx=0x7f787401ad40 comp rx=0 tx=0).stop 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 shutdown_connections 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7864066e90 0x7f7864069350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f788415a420 0x7f78840c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7884150c40 0x7f78840c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 --2- 172.21.15.32:0/544840491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78840725e0 0x7f788415cd40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 >> 172.21.15.32:0/544840491 conn(0x7f788406c0d0 msgr2=0x7f7884147950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 shutdown_connections 2024-03-16T19:56:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:30.928+0000 7f78889cb640 1 -- 172.21.15.32:0/544840491 wait complete. 2024-03-16T19:56:30.916 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T19:56:30.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T19:56:31.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 Processor -- start 2024-03-16T19:56:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 -- start start 2024-03-16T19:56:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f94bc0ca740 0x7f94bc0cab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc15de10 con 0x7f94bc0cba30 2024-03-16T19:56:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0c4ed0 con 0x7f94bc1541c0 2024-03-16T19:56:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94c1deb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc1545a0 con 0x7f94bc0ca740 2024-03-16T19:56:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94baffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f94bc0cba30 0x7f94bc159ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38076/0 (socket says 172.21.15.32:38076) 2024-03-16T19:56:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94baffd640 1 -- 172.21.15.32:0/2830243276 learned_addr learned my addr 172.21.15.32:0/2830243276 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:31.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.076+0000 7f94bbfff640 1 --2- 172.21.15.32:0/2830243276 >> v2:172.21.15.59:3300/0 conn(0x7f94bc0ca740 0x7f94bc0cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.080+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763193046 0 0) 0x7f94bc15de10 con 0x7f94bc0cba30 2024-03-16T19:56:31.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.080+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a8003530 con 0x7f94bc0cba30 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930151074 0 0) 0x7f94a8003530 con 0x7f94bc0cba30 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 >> v2:172.21.15.59:3300/0 conn(0x7f94bc0ca740 msgr2=0x7f94bc0cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 --2- 172.21.15.32:0/2830243276 >> v2:172.21.15.59:3300/0 conn(0x7f94bc0ca740 0x7f94bc0cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 >> v1:172.21.15.59:6789/0 conn(0x7f94bc1541c0 legacy=0x7f94bc0c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc15aa10 con 0x7f94bc0cba30 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f94a4001680 con 0x7f94bc0cba30 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 >> v1:172.21.15.32:6789/0 conn(0x7f94bc0cba30 legacy=0x7f94bc159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8003980 0x7f94a8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a80047d0 0x7f94a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 0x7f94a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1545a0 con 0x7f94a8005190 2024-03-16T19:56:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a800c0a0 con 0x7f94a80047d0 2024-03-16T19:56:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a800c260 con 0x7f94a8003980 2024-03-16T19:56:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.084+0000 7f94bb7fe640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8003980 0x7f94a8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.088+0000 7f94bbfff640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 0x7f94a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 -- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8003980 msgr2=0x7f94a8003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8003980 0x7f94a8003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 -- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a80047d0 msgr2=0x7f94a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94a80047d0 0x7f94a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 -- 172.21.15.32:0/2830243276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8003530 con 0x7f94a8005190 2024-03-16T19:56:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 -- 172.21.15.32:0/2830243276 reap_dead start 2024-03-16T19:56:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.092+0000 7f94bbfff640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 0x7f94a800b990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94b4007360 tx=0x7f94b401ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b129c94f9833825 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:31.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a5d80 con 0x7f94a8005190 2024-03-16T19:56:31.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b40039a0 con 0x7f94a8005190 2024-03-16T19:56:31.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94ba7fc640 1 -- 172.21.15.32:0/2830243276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4003d50 con 0x7f94a8005190 2024-03-16T19:56:31.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 msgr2=0x7f94a800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:31.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 0x7f94a800b990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94b4007360 tx=0x7f94b401ce80 comp rx=0 tx=0).stop 2024-03-16T19:56:31.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- 172.21.15.32:0/2830243276 shutdown_connections 2024-03-16T19:56:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 --2- 172.21.15.32:0/2830243276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8005190 0x7f94a800b990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- 172.21.15.32:0/2830243276 >> 172.21.15.32:0/2830243276 conn(0x7f94bc06c2a0 msgr2=0x7f94bc06cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- 172.21.15.32:0/2830243276 shutdown_connections 2024-03-16T19:56:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- 172.21.15.32:0/2830243276 wait complete. 2024-03-16T19:56:31.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 Processor -- start 2024-03-16T19:56:31.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- start start 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8005190 0x7f94bc0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc150590 0x7f94bc15c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1f3f50 con 0x7f94bc0c66c0 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1f3ac0 con 0x7f94bc150590 2024-03-16T19:56:31.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.100+0000 7f94c1deb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0c78c0 con 0x7f94a8005190 2024-03-16T19:56:31.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.104+0000 7f94bbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.104+0000 7f94bbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32980/0 (socket says 172.21.15.32:32980) 2024-03-16T19:56:31.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.104+0000 7f94bbfff640 1 -- 172.21.15.32:0/826872442 learned_addr learned my addr 172.21.15.32:0/826872442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:31.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.104+0000 7f94bb7fe640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8005190 0x7f94bc0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.104+0000 7f94baffd640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc150590 0x7f94bc15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 -- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8005190 msgr2=0x7f94bc0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:31.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8005190 0x7f94bc0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 -- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc150590 msgr2=0x7f94bc15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:31.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc150590 0x7f94bc15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:31.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 -- 172.21.15.32:0/826872442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc1f3190 con 0x7f94bc0c66c0 2024-03-16T19:56:31.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.108+0000 7f94bbfff640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f94b401ccb0 tx=0x7f94b4019c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b40ade00 con 0x7f94bc0c66c0 2024-03-16T19:56:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b40a5d90 con 0x7f94bc0c66c0 2024-03-16T19:56:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b40b5660 con 0x7f94bc0c66c0 2024-03-16T19:56:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc1f3470 con 0x7f94bc0c66c0 2024-03-16T19:56:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc0c75c0 con 0x7f94bc0c66c0 2024-03-16T19:56:31.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.112+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc06a340 con 0x7f94bc0c66c0 2024-03-16T19:56:31.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.404+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f94b40b57c0 con 0x7f94bc0c66c0 2024-03-16T19:56:31.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.408+0000 7f94b3fff640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 0x7f949c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:31.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.408+0000 7f94bb7fe640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 0x7f949c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:31.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.408+0000 7f94bb7fe640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 0x7f949c069380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94ac007fd0 tx=0x7f94ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:31.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.444+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f94b40f1140 con 0x7f94bc0c66c0 2024-03-16T19:56:31.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.444+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f94b40ed3e0 con 0x7f94bc0c66c0 2024-03-16T19:56:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:31.616+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f94bc1f3c00 con 0x7f94bc0c66c0 2024-03-16T19:56:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.748+0000 7f94b3fff640 1 -- 172.21.15.32:0/826872442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f94b40f0aa0 con 0x7f94bc0c66c0 2024-03-16T19:56:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 msgr2=0x7f949c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 0x7f949c069380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94ac007fd0 tx=0x7f94ac009040 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 msgr2=0x7f94bc1efaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f94b401ccb0 tx=0x7f94b4019c20 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 shutdown_connections 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f949c066ec0 0x7f949c069380 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94bc0c66c0 0x7f94bc1efaa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc150590 0x7f94bc15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 --2- 172.21.15.32:0/826872442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94a8005190 0x7f94bc0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 >> 172.21.15.32:0/826872442 conn(0x7f94bc06c2a0 msgr2=0x7f94bc0c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 shutdown_connections 2024-03-16T19:56:32.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.752+0000 7f94c1deb640 1 -- 172.21.15.32:0/826872442 wait complete. 2024-03-16T19:56:32.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-16T19:56:32.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 Processor -- start 2024-03-16T19:56:32.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 -- start start 2024-03-16T19:56:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb750072350 0x7fb7500730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb750154650 con 0x7fb750071910 2024-03-16T19:56:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7500c21d0 con 0x7fb750072730 2024-03-16T19:56:32.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb758722640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7500c83d0 con 0x7fb750072350 2024-03-16T19:56:32.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb7564be640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb750072350 0x7fb7500730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:32.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb755cbd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb750071910 0x7fb7500c1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38084/0 (socket says 172.21.15.32:38084) 2024-03-16T19:56:32.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.928+0000 7fb755cbd640 1 -- 172.21.15.32:0/2551691069 learned_addr learned my addr 172.21.15.32:0/2551691069 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.936+0000 7fb7564be640 1 -- 172.21.15.32:0/2551691069 >> v1:172.21.15.59:6789/0 conn(0x7fb750072730 legacy=0x7fb7501647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.936+0000 7fb7564be640 1 -- 172.21.15.32:0/2551691069 >> v1:172.21.15.32:6789/0 conn(0x7fb750071910 legacy=0x7fb7500c1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.936+0000 7fb7564be640 1 -- 172.21.15.32:0/2551691069 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75015e010 con 0x7fb750072350 2024-03-16T19:56:32.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.940+0000 7fb7564be640 1 --2- 172.21.15.32:0/2551691069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb7500730b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0097c0 tx=0x7fb74c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3293ff3c5a8cb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb7554bc640 1 -- 172.21.15.32:0/2551691069 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0a8950 con 0x7fb750072350 2024-03-16T19:56:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb7554bc640 1 -- 172.21.15.32:0/2551691069 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c0a8ab0 con 0x7fb750072350 2024-03-16T19:56:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb7554bc640 1 -- 172.21.15.32:0/2551691069 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0a8e40 con 0x7fb750072350 2024-03-16T19:56:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 -- 172.21.15.32:0/2551691069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 msgr2=0x7fb7500730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 --2- 172.21.15.32:0/2551691069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb7500730b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0097c0 tx=0x7fb74c01ec50 comp rx=0 tx=0).stop 2024-03-16T19:56:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 -- 172.21.15.32:0/2551691069 shutdown_connections 2024-03-16T19:56:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 --2- 172.21.15.32:0/2551691069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb7500730b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 -- 172.21.15.32:0/2551691069 >> 172.21.15.32:0/2551691069 conn(0x7fb75006f290 msgr2=0x7fb75006f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:32.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 -- 172.21.15.32:0/2551691069 shutdown_connections 2024-03-16T19:56:32.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.944+0000 7fb758722640 1 -- 172.21.15.32:0/2551691069 wait complete. 2024-03-16T19:56:32.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 Processor -- start 2024-03-16T19:56:32.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 -- start start 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb750071910 0x7fb7501e3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb750072730 0x7fb7501efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75015b240 con 0x7fb750072350 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7501e3780 con 0x7fb750071910 2024-03-16T19:56:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb758722640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75015d260 con 0x7fb750072730 2024-03-16T19:56:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb755cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb755cbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32992/0 (socket says 172.21.15.32:32992) 2024-03-16T19:56:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb755cbd640 1 -- 172.21.15.32:0/2195319319 learned_addr learned my addr 172.21.15.32:0/2195319319 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:32.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.948+0000 7fb756cbf640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb750072730 0x7fb7501efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.952+0000 7fb755cbd640 1 -- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb750072730 msgr2=0x7fb7501efcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.952+0000 7fb755cbd640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb750072730 0x7fb7501efcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.952+0000 7fb755cbd640 1 -- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb750071910 msgr2=0x7fb7501e3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.952+0000 7fb755cbd640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb750071910 0x7fb7501e3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.952+0000 7fb755cbd640 1 -- 172.21.15.32:0/2195319319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c009420 con 0x7fb750072350 2024-03-16T19:56:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:32.956+0000 7fb755cbd640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c420 tx=0x7fb74000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740005cf0 con 0x7fb750072350 2024-03-16T19:56:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740005e50 con 0x7fb750072350 2024-03-16T19:56:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400072f0 con 0x7fb750072350 2024-03-16T19:56:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7501f3300 con 0x7fb750072350 2024-03-16T19:56:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7501f38c0 con 0x7fb750072350 2024-03-16T19:56:33.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.040+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75006e550 con 0x7fb750072350 2024-03-16T19:56:33.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.344+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb740007450 con 0x7fb750072350 2024-03-16T19:56:33.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.344+0000 7fb746ffd640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 0x7fb734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:33.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.344+0000 7fb7564be640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 0x7fb734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:33.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.348+0000 7fb7564be640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 0x7fb734069300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0b2000 tx=0x7fb74c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.388+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb740055010 con 0x7fb750072350 2024-03-16T19:56:33.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.404+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb7400512b0 con 0x7fb750072350 2024-03-16T19:56:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:33.644+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fb7500c6c70 con 0x7fb750072350 2024-03-16T19:56:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.868+0000 7fb746ffd640 1 -- 172.21.15.32:0/2195319319 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7fb740017020 con 0x7fb750072350 2024-03-16T19:56:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-16T19:56:34.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 msgr2=0x7fb734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:34.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 0x7fb734069300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0b2000 tx=0x7fb74c004740 comp rx=0 tx=0).stop 2024-03-16T19:56:34.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 msgr2=0x7fb750158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:34.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c420 tx=0x7fb74000c8e0 comp rx=0 tx=0).stop 2024-03-16T19:56:34.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 shutdown_connections 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb734066e40 0x7fb734069300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb750072730 0x7fb7501efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb750072350 0x7fb750158f30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 --2- 172.21.15.32:0/2195319319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb750071910 0x7fb7501e3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 >> 172.21.15.32:0/2195319319 conn(0x7fb75006f290 msgr2=0x7fb750153910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.872+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 shutdown_connections 2024-03-16T19:56:34.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:34.876+0000 7fb758722640 1 -- 172.21.15.32:0/2195319319 wait complete. 2024-03-16T19:56:34.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T19:56:35.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 Processor -- start 2024-03-16T19:56:35.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 -- start start 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30281507e0 0x7f3028072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30280b1aa0 0x7f302815a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f3028072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3028072fb0 con 0x7f30281507e0 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280c2660 con 0x7f30280b1aa0 2024-03-16T19:56:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f302da71640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280c2940 con 0x7f3028071cc0 2024-03-16T19:56:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f30277fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30281507e0 0x7f3028072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f30277fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30281507e0 0x7f3028072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32996/0 (socket says 172.21.15.32:32996) 2024-03-16T19:56:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.060+0000 7f30277fe640 1 -- 172.21.15.32:0/3913429418 learned_addr learned my addr 172.21.15.32:0/3913429418 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.064+0000 7f30277fe640 1 -- 172.21.15.32:0/3913429418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 msgr2=0x7f3028072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.064+0000 7f30277fe640 1 --2- 172.21.15.32:0/3913429418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f3028072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.064+0000 7f30277fe640 1 -- 172.21.15.32:0/3913429418 >> v2:172.21.15.59:3300/0 conn(0x7f30280b1aa0 msgr2=0x7f302815a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.064+0000 7f30277fe640 1 --2- 172.21.15.32:0/3913429418 >> v2:172.21.15.59:3300/0 conn(0x7f30280b1aa0 0x7f302815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.064+0000 7f30277fe640 1 -- 172.21.15.32:0/3913429418 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280c2c20 con 0x7f30281507e0 2024-03-16T19:56:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.068+0000 7f30277fe640 1 --2- 172.21.15.32:0/3913429418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30281507e0 0x7f3028072a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f302806ea50 tx=0x7f301801fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe297ad0320d10b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.068+0000 7f30267fc640 1 -- 172.21.15.32:0/3913429418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180a9ae0 con 0x7f30281507e0 2024-03-16T19:56:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.068+0000 7f30267fc640 1 -- 172.21.15.32:0/3913429418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180a9c40 con 0x7f30281507e0 2024-03-16T19:56:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.068+0000 7f30267fc640 1 -- 172.21.15.32:0/3913429418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180b1440 con 0x7f30281507e0 2024-03-16T19:56:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- 172.21.15.32:0/3913429418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30281507e0 msgr2=0x7f3028072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- 172.21.15.32:0/3913429418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30281507e0 0x7f3028072a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f302806ea50 tx=0x7f301801fe20 comp rx=0 tx=0).stop 2024-03-16T19:56:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- 172.21.15.32:0/3913429418 shutdown_connections 2024-03-16T19:56:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- 172.21.15.32:0/3913429418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f3028072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- 172.21.15.32:0/3913429418 >> v2:172.21.15.59:3300/0 conn(0x7f30280b1aa0 0x7f302815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- 172.21.15.32:0/3913429418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30281507e0 0x7f3028072a70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- 172.21.15.32:0/3913429418 >> 172.21.15.32:0/3913429418 conn(0x7f302806f1a0 msgr2=0x7f302806f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- 172.21.15.32:0/3913429418 shutdown_connections 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- 172.21.15.32:0/3913429418 wait complete. 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 Processor -- start 2024-03-16T19:56:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- start start 2024-03-16T19:56:35.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f30280c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281507e0 0x7f30280aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280c4170 con 0x7f30280b1aa0 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280672c0 con 0x7f30281507e0 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f302da71640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302806c280 con 0x7f3028071cc0 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f3026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f3026ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33000/0 (socket says 172.21.15.32:33000) 2024-03-16T19:56:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.072+0000 7f3026ffd640 1 -- 172.21.15.32:0/1592959213 learned_addr learned my addr 172.21.15.32:0/1592959213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:35.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 -- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 msgr2=0x7f30280c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f30280c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 -- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281507e0 msgr2=0x7f30280aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281507e0 0x7f30280aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 -- 172.21.15.32:0/1592959213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30180094d0 con 0x7f30280b1aa0 2024-03-16T19:56:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.080+0000 7f3026ffd640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f301c00b550 tx=0x7f301c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c010870 con 0x7f30280b1aa0 2024-03-16T19:56:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c0109f0 con 0x7f30280b1aa0 2024-03-16T19:56:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c010da0 con 0x7f30280b1aa0 2024-03-16T19:56:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30281deec0 con 0x7f30280b1aa0 2024-03-16T19:56:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30281df160 con 0x7f30280b1aa0 2024-03-16T19:56:35.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.088+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30280c4340 con 0x7f30280b1aa0 2024-03-16T19:56:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.432+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f301c023480 con 0x7f30280b1aa0 2024-03-16T19:56:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.432+0000 7f300ffff640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 0x7f3014069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.436+0000 7f30277fe640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 0x7f3014069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.436+0000 7f30277fe640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 0x7f3014069300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30180b3000 tx=0x7f3018004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.476+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f301c055f50 con 0x7f30280b1aa0 2024-03-16T19:56:35.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.476+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f301c055620 con 0x7f30280b1aa0 2024-03-16T19:56:35.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:35.696+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3028068000 con 0x7f30280b1aa0 2024-03-16T19:56:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.020+0000 7f300ffff640 1 -- 172.21.15.32:0/1592959213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f301c01e020 con 0x7f30280b1aa0 2024-03-16T19:56:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 msgr2=0x7f3014069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 0x7f3014069300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30180b3000 tx=0x7f3018004740 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 msgr2=0x7f30280c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f301c00b550 tx=0x7f301c00ba10 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 shutdown_connections 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3014066e40 0x7f3014069300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30281507e0 0x7f30280aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30280b1aa0 0x7f30280c4b60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 --2- 172.21.15.32:0/1592959213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3028071cc0 0x7f30280c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 >> 172.21.15.32:0/1592959213 conn(0x7f302806f1a0 msgr2=0x7f30280b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 shutdown_connections 2024-03-16T19:56:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.024+0000 7f302da71640 1 -- 172.21.15.32:0/1592959213 wait complete. 2024-03-16T19:56:37.019 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-16T19:56:37.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-16T19:56:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.180+0000 7f35cfebf640 1 Processor -- start 2024-03-16T19:56:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.180+0000 7f35cfebf640 1 -- start start 2024-03-16T19:56:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cfebf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 0x7f35c8165750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cfebf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c806f990 con 0x7f35c815a7f0 2024-03-16T19:56:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cfebf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c8154270 con 0x7f35c8153e90 2024-03-16T19:56:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cfebf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80c3a70 con 0x7f35c806d0f0 2024-03-16T19:56:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cd45a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35c815a7f0 0x7f35c806c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38090/0 (socket says 172.21.15.32:38090) 2024-03-16T19:56:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.184+0000 7f35cd45a640 1 -- 172.21.15.32:0/2141253767 learned_addr learned my addr 172.21.15.32:0/2141253767 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.188+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310259713 0 0) 0x7f35c806f990 con 0x7f35c815a7f0 2024-03-16T19:56:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.188+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b4003530 con 0x7f35c815a7f0 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863814558 0 0) 0x7f35b4003530 con 0x7f35c815a7f0 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 msgr2=0x7f35c8165750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 0x7f35c8165750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 >> v1:172.21.15.59:6789/0 conn(0x7f35c8153e90 legacy=0x7f35c80c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80c81e0 con 0x7f35c815a7f0 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f35b8001680 con 0x7f35c815a7f0 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 >> v1:172.21.15.32:6789/0 conn(0x7f35c815a7f0 legacy=0x7f35c806c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 0x7f35b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35b40040f0 0x7f35b40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35b400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80c3a70 con 0x7f35b4004e30 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40055d0 con 0x7f35b40040f0 2024-03-16T19:56:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.192+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b400bed0 con 0x7f35c806d0f0 2024-03-16T19:56:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.196+0000 7f35cdc5b640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 0x7f35b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.196+0000 7f35ce45c640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35b400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35cd45a640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35b40040f0 0x7f35b40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35ce45c640 1 -- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 msgr2=0x7f35b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35ce45c640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c806d0f0 0x7f35b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35ce45c640 1 -- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35b40040f0 msgr2=0x7f35b40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35ce45c640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35b40040f0 0x7f35b40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.200+0000 7f35ce45c640 1 -- 172.21.15.32:0/2141253767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4003530 con 0x7f35b4004e30 2024-03-16T19:56:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.204+0000 7f35ce45c640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35b400b990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009ca0 tx=0x7f35c001cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e8e1029e775823 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:37.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00a5c40 con 0x7f35b4004e30 2024-03-16T19:56:37.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c00a5dc0 con 0x7f35b4004e30 2024-03-16T19:56:37.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35ccc59640 1 -- 172.21.15.32:0/2141253767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00ad680 con 0x7f35b4004e30 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- 172.21.15.32:0/2141253767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 msgr2=0x7f35b400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 --2- 172.21.15.32:0/2141253767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35b400b990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009ca0 tx=0x7f35c001cc30 comp rx=0 tx=0).stop 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35ce45c640 1 -- 172.21.15.32:0/2141253767 reap_dead start 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- 172.21.15.32:0/2141253767 shutdown_connections 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- 172.21.15.32:0/2141253767 >> 172.21.15.32:0/2141253767 conn(0x7f35c8070f90 msgr2=0x7f35c80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- 172.21.15.32:0/2141253767 shutdown_connections 2024-03-16T19:56:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- 172.21.15.32:0/2141253767 wait complete. 2024-03-16T19:56:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 Processor -- start 2024-03-16T19:56:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.212+0000 7f35cfebf640 1 -- start start 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c80c60e0 0x7f35c814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c814f730 0x7f35c80c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c814fbb0 con 0x7f35b4004e30 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80c6630 con 0x7f35c80c60e0 2024-03-16T19:56:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cfebf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80c6910 con 0x7f35c814f730 2024-03-16T19:56:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cdc5b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35ce45c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c814f730 0x7f35c80c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cdc5b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33010/0 (socket says 172.21.15.32:33010) 2024-03-16T19:56:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.216+0000 7f35cdc5b640 1 -- 172.21.15.32:0/4255736365 learned_addr learned my addr 172.21.15.32:0/4255736365 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:37.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cd45a640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c80c60e0 0x7f35c814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cdc5b640 1 -- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c814f730 msgr2=0x7f35c80c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cdc5b640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c814f730 0x7f35c80c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cdc5b640 1 -- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c80c60e0 msgr2=0x7f35c814f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cdc5b640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c80c60e0 0x7f35c814f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.220+0000 7f35cdc5b640 1 -- 172.21.15.32:0/4255736365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8155190 con 0x7f35b4004e30 2024-03-16T19:56:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.224+0000 7f35cdc5b640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f35c400e4b0 tx=0x7f35c400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40052e0 con 0x7f35b4004e30 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4005460 con 0x7f35b4004e30 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4003840 con 0x7f35b4004e30 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c8157da0 con 0x7f35b4004e30 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c81efca0 con 0x7f35b4004e30 2024-03-16T19:56:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.240+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c806ff90 con 0x7f35b4004e30 2024-03-16T19:56:37.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.604+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f35c40039a0 con 0x7f35b4004e30 2024-03-16T19:56:37.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.604+0000 7f35be7fc640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 0x7f35ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:37.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.604+0000 7f35cd45a640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 0x7f35ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.608+0000 7f35cd45a640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 0x7f35ac069300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b3ac0 tx=0x7f35b8001b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:37.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.628+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35c40564e0 con 0x7f35b4004e30 2024-03-16T19:56:37.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.628+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f35c4052080 con 0x7f35b4004e30 2024-03-16T19:56:37.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:37.876+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f35c8068000 con 0x7f35b4004e30 2024-03-16T19:56:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.840+0000 7f35be7fc640 1 -- 172.21.15.32:0/4255736365 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v24) v1 ==== 157+0+0 (secure 0 0 0) 0x7f35c4055740 con 0x7f35b4004e30 2024-03-16T19:56:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 msgr2=0x7f35ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 0x7f35ac069300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b3ac0 tx=0x7f35b8001b80 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 msgr2=0x7f35c80c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f35c400e4b0 tx=0x7f35c400e970 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 shutdown_connections 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f35ac066e40 0x7f35ac069300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35c814f730 0x7f35c80c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35c80c60e0 0x7f35c814f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 --2- 172.21.15.32:0/4255736365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35b4004e30 0x7f35c80c5a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 >> 172.21.15.32:0/4255736365 conn(0x7f35c8070f90 msgr2=0x7f35c806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 shutdown_connections 2024-03-16T19:56:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:38.844+0000 7f35cfebf640 1 -- 172.21.15.32:0/4255736365 wait complete. 2024-03-16T19:56:38.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T19:56:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.000+0000 7feb555c6640 1 Processor -- start 2024-03-16T19:56:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.000+0000 7feb555c6640 1 -- start start 2024-03-16T19:56:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb555c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feb50150790 0x7feb5015a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb555c6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500c5fc0 con 0x7feb500c7c90 2024-03-16T19:56:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb555c6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500c6860 con 0x7feb500cbe50 2024-03-16T19:56:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb555c6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5015a7c0 con 0x7feb50150790 2024-03-16T19:56:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb4f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb500c7c90 0x7feb50160c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38104/0 (socket says 172.21.15.32:38104) 2024-03-16T19:56:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb4f7fe640 1 -- 172.21.15.32:0/2472053943 learned_addr learned my addr 172.21.15.32:0/2472053943 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.004+0000 7feb4effd640 1 --2- 172.21.15.32:0/2472053943 >> v2:172.21.15.59:3300/0 conn(0x7feb50150790 0x7feb5015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.008+0000 7feb4dffb640 1 -- 172.21.15.32:0/2472053943 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666487673 0 0) 0x7feb500c5fc0 con 0x7feb500c7c90 2024-03-16T19:56:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.008+0000 7feb4dffb640 1 -- 172.21.15.32:0/2472053943 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38004640 con 0x7feb500c7c90 2024-03-16T19:56:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.012+0000 7feb4effd640 1 -- 172.21.15.32:0/2472053943 >> v1:172.21.15.59:6789/0 conn(0x7feb500cbe50 legacy=0x7feb500c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.012+0000 7feb4effd640 1 -- 172.21.15.32:0/2472053943 >> v1:172.21.15.32:6789/0 conn(0x7feb500c7c90 legacy=0x7feb50160c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.012+0000 7feb4effd640 1 -- 172.21.15.32:0/2472053943 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500c4220 con 0x7feb50150790 2024-03-16T19:56:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.020+0000 7feb4effd640 1 --2- 172.21.15.32:0/2472053943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb50150790 0x7feb5015a280 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb400097c0 tx=0x7feb4001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=823fb62f40a707bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb4dffb640 1 -- 172.21.15.32:0/2472053943 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400a8950 con 0x7feb50150790 2024-03-16T19:56:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb4dffb640 1 -- 172.21.15.32:0/2472053943 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb400a8ab0 con 0x7feb50150790 2024-03-16T19:56:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb4dffb640 1 -- 172.21.15.32:0/2472053943 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400a8e40 con 0x7feb50150790 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 -- 172.21.15.32:0/2472053943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb50150790 msgr2=0x7feb5015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 --2- 172.21.15.32:0/2472053943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb50150790 0x7feb5015a280 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb400097c0 tx=0x7feb4001ec50 comp rx=0 tx=0).stop 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 -- 172.21.15.32:0/2472053943 shutdown_connections 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 --2- 172.21.15.32:0/2472053943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb50150790 0x7feb5015a280 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 -- 172.21.15.32:0/2472053943 >> 172.21.15.32:0/2472053943 conn(0x7feb5006ed70 msgr2=0x7feb5006f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 -- 172.21.15.32:0/2472053943 shutdown_connections 2024-03-16T19:56:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.036+0000 7feb555c6640 1 -- 172.21.15.32:0/2472053943 wait complete. 2024-03-16T19:56:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 Processor -- start 2024-03-16T19:56:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 -- start start 2024-03-16T19:56:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb500c7c90 0x7feb500bf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feb500cbe50 0x7feb5014ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb501f1d70 con 0x7feb50150790 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb501f0e80 con 0x7feb500c7c90 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb555c6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb501f1160 con 0x7feb500cbe50 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb4f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb4f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33016/0 (socket says 172.21.15.32:33016) 2024-03-16T19:56:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.040+0000 7feb4f7fe640 1 -- 172.21.15.32:0/1230092946 learned_addr learned my addr 172.21.15.32:0/1230092946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:39.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.044+0000 7feb4f7fe640 1 -- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feb500cbe50 msgr2=0x7feb5014ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:39.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.044+0000 7feb4f7fe640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feb500cbe50 0x7feb5014ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:39.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.044+0000 7feb4f7fe640 1 -- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb500c7c90 msgr2=0x7feb500bf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:39.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.044+0000 7feb4f7fe640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb500c7c90 0x7feb500bf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:39.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.044+0000 7feb4f7fe640 1 -- 172.21.15.32:0/1230092946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009420 con 0x7feb50150790 2024-03-16T19:56:39.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.048+0000 7feb4f7fe640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cb40 tx=0x7feb4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4800aca0 con 0x7feb50150790 2024-03-16T19:56:39.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4800ae00 con 0x7feb50150790 2024-03-16T19:56:39.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48012620 con 0x7feb50150790 2024-03-16T19:56:39.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb501f1440 con 0x7feb50150790 2024-03-16T19:56:39.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb501f1a00 con 0x7feb50150790 2024-03-16T19:56:39.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.056+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5006a340 con 0x7feb50150790 2024-03-16T19:56:39.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.076+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feb48007070 con 0x7feb50150790 2024-03-16T19:56:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.080+0000 7feb377fe640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 0x7feb3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.080+0000 7feb4effd640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 0x7feb3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.080+0000 7feb4effd640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 0x7feb3c069300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb400b2000 tx=0x7feb40004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:39.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.416+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6007+0+0 (secure 0 0 0) 0x7feb48056150 con 0x7feb50150790 2024-03-16T19:56:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.416+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7feb48051cf0 con 0x7feb50150790 2024-03-16T19:56:39.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:39.600+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7feb501f1eb0 con 0x7feb50150790 2024-03-16T19:56:41.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:56:41.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:56:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb377fe640 1 -- 172.21.15.32:0/1230092946 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v26) v1 ==== 228+0+0 (secure 0 0 0) 0x7feb48017070 con 0x7feb50150790 2024-03-16T19:56:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data_ec' created 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 msgr2=0x7feb3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 0x7feb3c069300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb400b2000 tx=0x7feb40004740 comp rx=0 tx=0).stop 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 msgr2=0x7feb501f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cb40 tx=0x7feb4800a060 comp rx=0 tx=0).stop 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 shutdown_connections 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feb3c066e40 0x7feb3c069300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb50150790 0x7feb501f2200 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feb500cbe50 0x7feb5014ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 --2- 172.21.15.32:0/1230092946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb500c7c90 0x7feb500bf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 >> 172.21.15.32:0/1230092946 conn(0x7feb5006ed70 msgr2=0x7feb500720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 shutdown_connections 2024-03-16T19:56:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.012+0000 7feb555c6640 1 -- 172.21.15.32:0/1230092946 wait complete. 2024-03-16T19:56:42.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-16T19:56:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 Processor -- start 2024-03-16T19:56:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 -- start start 2024-03-16T19:56:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 0x7f94e0072960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e006d050 con 0x7f94e0072580 2024-03-16T19:56:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e00c6820 con 0x7f94e006e9d0 2024-03-16T19:56:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.212+0000 7f94e5eb0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00c7f20 con 0x7f94e0074380 2024-03-16T19:56:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.216+0000 7f94dffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f94e006e9d0 0x7f94e0072ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44434/0 (socket says 172.21.15.32:44434) 2024-03-16T19:56:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.216+0000 7f94dffff640 1 -- 172.21.15.32:0/119287891 learned_addr learned my addr 172.21.15.32:0/119287891 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.220+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754167771 0 0) 0x7f94e00c6820 con 0x7f94e006e9d0 2024-03-16T19:56:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.220+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c8003530 con 0x7f94e006e9d0 2024-03-16T19:56:42.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.224+0000 7f94deffd640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 0x7f94e0072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1451817100 0 0) 0x7f94c8003530 con 0x7f94e006e9d0 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 msgr2=0x7f94e0072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 0x7f94e0072960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 >> v1:172.21.15.32:6789/0 conn(0x7f94e0072580 legacy=0x7f94e0158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0157b60 con 0x7f94e006e9d0 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f94d8001a00 con 0x7f94e006e9d0 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 >> v1:172.21.15.59:6789/0 conn(0x7f94e006e9d0 legacy=0x7f94e0072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 0x7f94c8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c8004d10 0x7f94c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00c7f20 con 0x7f94c8003f30 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c800bb00 con 0x7f94c8004d10 2024-03-16T19:56:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.228+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c800bcc0 con 0x7f94e0074380 2024-03-16T19:56:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.232+0000 7f94deffd640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.236+0000 7f94deffd640 1 -- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 msgr2=0x7f94c8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.236+0000 7f94deffd640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e0074380 0x7f94c8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.236+0000 7f94deffd640 1 -- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c8004d10 msgr2=0x7f94c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.236+0000 7f94deffd640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c8004d10 0x7f94c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.236+0000 7f94deffd640 1 -- 172.21.15.32:0/119287891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8003530 con 0x7f94c8003f30 2024-03-16T19:56:42.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.240+0000 7f94deffd640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94c80047d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009ae0 tx=0x7f94d401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595ac68b47d82914 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.244+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d40a56a0 con 0x7f94c8003f30 2024-03-16T19:56:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.244+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d40a5800 con 0x7f94c8003f30 2024-03-16T19:56:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.244+0000 7f94de7fc640 1 -- 172.21.15.32:0/119287891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d40a5bf0 con 0x7f94c8003f30 2024-03-16T19:56:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/119287891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 msgr2=0x7f94c80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/119287891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94c80047d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009ae0 tx=0x7f94d401cab0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94dffff640 1 -- 172.21.15.32:0/119287891 reap_dead start 2024-03-16T19:56:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/119287891 shutdown_connections 2024-03-16T19:56:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/119287891 >> 172.21.15.32:0/119287891 conn(0x7f94e006f360 msgr2=0x7f94e006f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/119287891 shutdown_connections 2024-03-16T19:56:42.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/119287891 wait complete. 2024-03-16T19:56:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 Processor -- start 2024-03-16T19:56:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- start start 2024-03-16T19:56:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e014e160 0x7f94e01f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e01f2f00 0x7f94e013bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00c1530 con 0x7f94c8003f30 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00c2990 con 0x7f94e01f2f00 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94e5eb0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e01552d0 con 0x7f94e014e160 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36340/0 (socket says 172.21.15.32:36340) 2024-03-16T19:56:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.248+0000 7f94df7fe640 1 -- 172.21.15.32:0/1625575948 learned_addr learned my addr 172.21.15.32:0/1625575948 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.252+0000 7f94deffd640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e014e160 0x7f94e01f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.252+0000 7f94dffff640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e01f2f00 0x7f94e013bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.256+0000 7f94df7fe640 1 -- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e014e160 msgr2=0x7f94e01f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.256+0000 7f94df7fe640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e014e160 0x7f94e01f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.256+0000 7f94df7fe640 1 -- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e01f2f00 msgr2=0x7f94e013bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.256+0000 7f94df7fe640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e01f2f00 0x7f94e013bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.256+0000 7f94df7fe640 1 -- 172.21.15.32:0/1625575948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00c0ee0 con 0x7f94c8003f30 2024-03-16T19:56:42.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.260+0000 7f94df7fe640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c480 tx=0x7f94d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0005480 con 0x7f94c8003f30 2024-03-16T19:56:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d0003600 con 0x7f94c8003f30 2024-03-16T19:56:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d00039d0 con 0x7f94c8003f30 2024-03-16T19:56:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e00c11c0 con 0x7f94c8003f30 2024-03-16T19:56:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e014c950 con 0x7f94c8003f30 2024-03-16T19:56:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.264+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e006a340 con 0x7f94c8003f30 2024-03-16T19:56:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.540+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f94d001b3b0 con 0x7f94c8003f30 2024-03-16T19:56:42.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.540+0000 7f94c7fff640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 0x7f94cc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:42.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.540+0000 7f94deffd640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 0x7f94cc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:42.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.540+0000 7f94deffd640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 0x7f94cc069380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94d40af000 tx=0x7f94d40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.564+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f94d0055060 con 0x7f94c8003f30 2024-03-16T19:56:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.564+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f94d0051300 con 0x7f94c8003f30 2024-03-16T19:56:42.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:42.752+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f94e006e7b0 con 0x7f94c8003f30 2024-03-16T19:56:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.244+0000 7f94c7fff640 1 -- 172.21.15.32:0/1625575948 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f94d00170c0 con 0x7f94c8003f30 2024-03-16T19:56:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 4 allow_ec_overwrites to true 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 msgr2=0x7f94cc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 0x7f94cc069380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94d40af000 tx=0x7f94d40046d0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 msgr2=0x7f94e014c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c480 tx=0x7f94d000c940 comp rx=0 tx=0).stop 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 shutdown_connections 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f94cc066ec0 0x7f94cc069380 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e01f2f00 0x7f94e013bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f94e014e160 0x7f94e01f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 --2- 172.21.15.32:0/1625575948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c8003f30 0x7f94e014c410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 >> 172.21.15.32:0/1625575948 conn(0x7f94e006f360 msgr2=0x7f94e006d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 shutdown_connections 2024-03-16T19:56:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.248+0000 7f94e5eb0640 1 -- 172.21.15.32:0/1625575948 wait complete. 2024-03-16T19:56:43.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-16T19:56:43.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 Processor -- start 2024-03-16T19:56:43.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 -- start start 2024-03-16T19:56:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f815a1d0 0x7f08f80c6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08f8072bd0 0x7f08f806c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80c5760 con 0x7f08f8151020 2024-03-16T19:56:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f80c6ac0 con 0x7f08f8072bd0 2024-03-16T19:56:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.384+0000 7f08fd58d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8151400 con 0x7f08f815a1d0 2024-03-16T19:56:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.424+0000 7f08f77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08f8072bd0 0x7f08f806c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.428+0000 7f08f6ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f08f8151020 0x7f08f8156c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44450/0 (socket says 172.21.15.32:44450) 2024-03-16T19:56:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.428+0000 7f08f6ffd640 1 -- 172.21.15.32:0/3221573664 learned_addr learned my addr 172.21.15.32:0/3221573664 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.440+0000 7f08f77fe640 1 -- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f815a1d0 msgr2=0x7f08f80c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.440+0000 7f08f77fe640 1 --2- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f815a1d0 0x7f08f80c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.440+0000 7f08f77fe640 1 -- 172.21.15.32:0/3221573664 >> v1:172.21.15.59:6789/0 conn(0x7f08f8151020 legacy=0x7f08f8156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.440+0000 7f08f77fe640 1 -- 172.21.15.32:0/3221573664 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80c5b10 con 0x7f08f8072bd0 2024-03-16T19:56:43.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.448+0000 7f08f77fe640 1 --2- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8072bd0 0x7f08f806c3d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009d90 tx=0x7f08e801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c163b9316e9af205 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.496+0000 7f08f5ffb640 1 -- 172.21.15.32:0/3221573664 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e80a5830 con 0x7f08f8072bd0 2024-03-16T19:56:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.496+0000 7f08f5ffb640 1 -- 172.21.15.32:0/3221573664 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e80a5990 con 0x7f08f8072bd0 2024-03-16T19:56:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.496+0000 7f08f5ffb640 1 -- 172.21.15.32:0/3221573664 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e80a5d20 con 0x7f08f8072bd0 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8072bd0 msgr2=0x7f08f806c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8072bd0 0x7f08f806c3d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009d90 tx=0x7f08e801c970 comp rx=0 tx=0).stop 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- 172.21.15.32:0/3221573664 shutdown_connections 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8072bd0 0x7f08f806c3d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- 172.21.15.32:0/3221573664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f815a1d0 0x7f08f80c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- 172.21.15.32:0/3221573664 >> 172.21.15.32:0/3221573664 conn(0x7f08f80cc7a0 msgr2=0x7f08f80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- 172.21.15.32:0/3221573664 shutdown_connections 2024-03-16T19:56:43.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- 172.21.15.32:0/3221573664 wait complete. 2024-03-16T19:56:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 Processor -- start 2024-03-16T19:56:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- start start 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f8072bd0 0x7f08f80cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08f815a1d0 0x7f08f8143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8158a70 con 0x7f08f815a1d0 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f814f3a0 con 0x7f08f8151020 2024-03-16T19:56:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.500+0000 7f08fd58d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f814f680 con 0x7f08f8072bd0 2024-03-16T19:56:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.504+0000 7f08f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.508+0000 7f08f6ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f8072bd0 0x7f08f80cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.508+0000 7f08f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:46298/0 (socket says 172.21.15.32:46298) 2024-03-16T19:56:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.508+0000 7f08f67fc640 1 -- 172.21.15.32:0/2876194492 learned_addr learned my addr 172.21.15.32:0/2876194492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.516+0000 7f08f67fc640 1 -- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f8072bd0 msgr2=0x7f08f80cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.516+0000 7f08f67fc640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f8072bd0 0x7f08f80cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.516+0000 7f08f67fc640 1 -- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08f815a1d0 msgr2=0x7f08f8143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.516+0000 7f08f67fc640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08f815a1d0 0x7f08f8143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.516+0000 7f08f67fc640 1 -- 172.21.15.32:0/2876194492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8009a70 con 0x7f08f8151020 2024-03-16T19:56:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.524+0000 7f08f67fc640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08e000b410 tx=0x7f08e000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.528+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0011830 con 0x7f08f8151020 2024-03-16T19:56:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.528+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e0011990 con 0x7f08f8151020 2024-03-16T19:56:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.528+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0011d40 con 0x7f08f8151020 2024-03-16T19:56:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.528+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f8073320 con 0x7f08f8151020 2024-03-16T19:56:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.528+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f814fc40 con 0x7f08f8151020 2024-03-16T19:56:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.532+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f8068000 con 0x7f08f8151020 2024-03-16T19:56:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.836+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08e001a400 con 0x7f08f8151020 2024-03-16T19:56:43.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.836+0000 7f08ef7fe640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 0x7f08d8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:43.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.840+0000 7f08f6ffd640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 0x7f08d8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.840+0000 7f08f6ffd640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 0x7f08d8069350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08f0000f70 tx=0x7f08f0007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.844+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f08e00562f0 con 0x7f08f8151020 2024-03-16T19:56:43.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:43.868+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f08e0051e90 con 0x7f08f8151020 2024-03-16T19:56:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:44.256+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f08f8154910 con 0x7f08f8151020 2024-03-16T19:56:45.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.532+0000 7f08ef7fe640 1 -- 172.21.15.32:0/2876194492 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f08f8154910 con 0x7f08f8151020 2024-03-16T19:56:45.506 INFO:teuthology.orchestra.run.smithi032.stderr:added data pool 4 to fsmap 2024-03-16T19:56:45.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 msgr2=0x7f08d8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 0x7f08d8069350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08f0000f70 tx=0x7f08f0007e90 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 msgr2=0x7f08f8160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08e000b410 tx=0x7f08e000b8d0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 shutdown_connections 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08d8066e90 0x7f08d8069350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08f815a1d0 0x7f08f8143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f8151020 0x7f08f8160000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 --2- 172.21.15.32:0/2876194492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08f8072bd0 0x7f08f80cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 >> 172.21.15.32:0/2876194492 conn(0x7f08f80cc7a0 msgr2=0x7f08f814e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 shutdown_connections 2024-03-16T19:56:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.536+0000 7f08fd58d640 1 -- 172.21.15.32:0/2876194492 wait complete. 2024-03-16T19:56:45.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.664+0000 7f8afcfbf640 1 Processor -- start 2024-03-16T19:56:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 -- start start 2024-03-16T19:56:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af80ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8af80caab0 0x7f8af8164740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af80cbfa0 con 0x7f8af80cc590 2024-03-16T19:56:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af815e2c0 con 0x7f8af80caab0 2024-03-16T19:56:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8afcfbf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af815a140 con 0x7f8af80cc970 2024-03-16T19:56:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8af6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8af80caab0 0x7f8af8164740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8af6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8af80caab0 0x7f8af8164740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36364/0 (socket says 172.21.15.32:36364) 2024-03-16T19:56:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8af6d9d640 1 -- 172.21.15.32:0/1651444192 learned_addr learned my addr 172.21.15.32:0/1651444192 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.668+0000 7f8af5d9b640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af80ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:45.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.672+0000 7f8af6d9d640 1 -- 172.21.15.32:0/1651444192 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 msgr2=0x7f8af80ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:45.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.672+0000 7f8af6d9d640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af80ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.672+0000 7f8af6d9d640 1 -- 172.21.15.32:0/1651444192 >> v1:172.21.15.59:6789/0 conn(0x7f8af80cc590 legacy=0x7f8af815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:45.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.672+0000 7f8af6d9d640 1 -- 172.21.15.32:0/1651444192 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af815a3e0 con 0x7f8af80caab0 2024-03-16T19:56:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.676+0000 7f8af6d9d640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80caab0 0x7f8af8164740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009e50 tx=0x7f8af001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3220309ef6667f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:45.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.680+0000 7f8af559a640 1 -- 172.21.15.32:0/1651444192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af00af400 con 0x7f8af80caab0 2024-03-16T19:56:45.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.680+0000 7f8af559a640 1 -- 172.21.15.32:0/1651444192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af00ab050 con 0x7f8af80caab0 2024-03-16T19:56:45.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.680+0000 7f8af559a640 1 -- 172.21.15.32:0/1651444192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af00af780 con 0x7f8af80caab0 2024-03-16T19:56:45.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.680+0000 7f8afcfbf640 1 -- 172.21.15.32:0/1651444192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80caab0 msgr2=0x7f8af8164740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:45.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.680+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80caab0 0x7f8af8164740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009e50 tx=0x7f8af001f1e0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- 172.21.15.32:0/1651444192 shutdown_connections 2024-03-16T19:56:45.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80caab0 0x7f8af8164740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/1651444192 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af80ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- 172.21.15.32:0/1651444192 >> 172.21.15.32:0/1651444192 conn(0x7f8af8066ea0 msgr2=0x7f8af80672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- 172.21.15.32:0/1651444192 shutdown_connections 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- 172.21.15.32:0/1651444192 wait complete. 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 Processor -- start 2024-03-16T19:56:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- start start 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8af80caab0 0x7f8af80c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af8151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af80c8b30 con 0x7f8af80cc590 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8151fb0 con 0x7f8af80caab0 2024-03-16T19:56:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8afcfbf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8152e40 con 0x7f8af80cc970 2024-03-16T19:56:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8af5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8af5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36376/0 (socket says 172.21.15.32:36376) 2024-03-16T19:56:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8af5d9b640 1 -- 172.21.15.32:0/3208424462 learned_addr learned my addr 172.21.15.32:0/3208424462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.684+0000 7f8af6d9d640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af8151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.692+0000 7f8af5d9b640 1 -- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 msgr2=0x7f8af8151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.692+0000 7f8af5d9b640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af8151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.692+0000 7f8af5d9b640 1 -- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8af80caab0 msgr2=0x7f8af80c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.692+0000 7f8af5d9b640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8af80caab0 0x7f8af80c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.692+0000 7f8af5d9b640 1 -- 172.21.15.32:0/3208424462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0009b30 con 0x7f8af80cc590 2024-03-16T19:56:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.696+0000 7f8af5d9b640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00e930 tx=0x7f8aec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.704+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec009690 con 0x7f8af80cc590 2024-03-16T19:56:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.704+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec00ca00 con 0x7f8af80cc590 2024-03-16T19:56:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.704+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af81530e0 con 0x7f8af80cc590 2024-03-16T19:56:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.704+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00cd70 con 0x7f8af80cc590 2024-03-16T19:56:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.704+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af815cda0 con 0x7f8af80cc590 2024-03-16T19:56:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:45.708+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af806a340 con 0x7f8af80cc590 2024-03-16T19:56:46.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.096+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8aec013470 con 0x7f8af80cc590 2024-03-16T19:56:46.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.096+0000 7f8ae6ffd640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 0x7f8ad8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.096+0000 7f8af659c640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 0x7f8ad8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:46.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.096+0000 7f8af659c640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 0x7f8ad8069350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8007fd0 tx=0x7f8ae80079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.136+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8aec056040 con 0x7f8af80cc590 2024-03-16T19:56:46.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.140+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8aec0522e0 con 0x7f8af80cc590 2024-03-16T19:56:46.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.344+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8af8153bb0 con 0x7f8af80cc590 2024-03-16T19:56:46.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.408+0000 7f8ae6ffd640 1 -- 172.21.15.32:0/3208424462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6550 (secure 0 0 0) 0x7f8aec021070 con 0x7f8af80cc590 2024-03-16T19:56:46.380 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:46.380 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:56:44.497940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4255,"qdb_cluster":[4255]},"id":1}]} 2024-03-16T19:56:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 msgr2=0x7f8ad8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 0x7f8ad8069350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8007fd0 tx=0x7f8ae80079a0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 msgr2=0x7f8af815bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00e930 tx=0x7f8aec00edf0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 shutdown_connections 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8ad8066e90 0x7f8ad8069350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8af80cc970 0x7f8af8151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8af80cc590 0x7f8af815bdb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 --2- 172.21.15.32:0/3208424462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8af80caab0 0x7f8af80c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 >> 172.21.15.32:0/3208424462 conn(0x7f8af8066ea0 msgr2=0x7f8af815dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 shutdown_connections 2024-03-16T19:56:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.412+0000 7f8afcfbf640 1 -- 172.21.15.32:0/3208424462 wait complete. 2024-03-16T19:56:46.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.536+0000 7f30187c2640 1 Processor -- start 2024-03-16T19:56:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.540+0000 7f30187c2640 1 -- start start 2024-03-16T19:56:46.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.540+0000 7f30187c2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 0x7f30100c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.540+0000 7f30187c2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30100c3940 con 0x7f30100c3e30 2024-03-16T19:56:46.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.540+0000 7f30187c2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3010072cc0 con 0x7f30100c4210 2024-03-16T19:56:46.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.540+0000 7f30187c2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30100c7ca0 con 0x7f3010151220 2024-03-16T19:56:46.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.544+0000 7f3015d5d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 0x7f30100c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:46.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.544+0000 7f3016d5f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f30100c4210 0x7f30100c8630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44466/0 (socket says 172.21.15.32:44466) 2024-03-16T19:56:46.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.544+0000 7f3016d5f640 1 -- 172.21.15.32:0/1144798439 learned_addr learned my addr 172.21.15.32:0/1144798439 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.552+0000 7f3015d5d640 1 -- 172.21.15.32:0/1144798439 >> v1:172.21.15.59:6789/0 conn(0x7f30100c4210 legacy=0x7f30100c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.552+0000 7f3015d5d640 1 -- 172.21.15.32:0/1144798439 >> v1:172.21.15.32:6789/0 conn(0x7f30100c3e30 legacy=0x7f3010150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.552+0000 7f3015d5d640 1 -- 172.21.15.32:0/1144798439 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100c7780 con 0x7f3010151220 2024-03-16T19:56:46.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.556+0000 7f3015d5d640 1 --2- 172.21.15.32:0/1144798439 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 0x7f30100c80f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3000009820 tx=0x7f300001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3903c7756a29863f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:46.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f301555c640 1 -- 172.21.15.32:0/1144798439 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000ab600 con 0x7f3010151220 2024-03-16T19:56:46.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f301555c640 1 -- 172.21.15.32:0/1144798439 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30000a7050 con 0x7f3010151220 2024-03-16T19:56:46.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f301555c640 1 -- 172.21.15.32:0/1144798439 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000ab980 con 0x7f3010151220 2024-03-16T19:56:46.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- 172.21.15.32:0/1144798439 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 msgr2=0x7f30100c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 --2- 172.21.15.32:0/1144798439 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 0x7f30100c80f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3000009820 tx=0x7f300001c480 comp rx=0 tx=0).stop 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- 172.21.15.32:0/1144798439 shutdown_connections 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 --2- 172.21.15.32:0/1144798439 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3010151220 0x7f30100c80f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- 172.21.15.32:0/1144798439 >> 172.21.15.32:0/1144798439 conn(0x7f301006e6c0 msgr2=0x7f301006c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- 172.21.15.32:0/1144798439 shutdown_connections 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- 172.21.15.32:0/1144798439 wait complete. 2024-03-16T19:56:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 Processor -- start 2024-03-16T19:56:46.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.572+0000 7f30187c2640 1 -- start start 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30100c4210 0x7f3010143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010151220 0x7f301014aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3010150530 con 0x7f3010151220 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301014ec30 con 0x7f30100c4210 2024-03-16T19:56:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f30187c2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3010146130 con 0x7f30100c3e30 2024-03-16T19:56:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f301655e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f3015d5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30100c4210 0x7f3010143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f301655e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:40258/0 (socket says 172.21.15.32:40258) 2024-03-16T19:56:46.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.576+0000 7f301655e640 1 -- 172.21.15.32:0/3325417498 learned_addr learned my addr 172.21.15.32:0/3325417498 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:46.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.588+0000 7f301655e640 1 -- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30100c4210 msgr2=0x7f3010143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.588+0000 7f301655e640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30100c4210 0x7f3010143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.588+0000 7f301655e640 1 -- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010151220 msgr2=0x7f301014aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.588+0000 7f301655e640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010151220 0x7f301014aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.588+0000 7f301655e640 1 -- 172.21.15.32:0/3325417498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009420 con 0x7f30100c3e30 2024-03-16T19:56:46.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.592+0000 7f301655e640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f300c00c490 tx=0x7f300c001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:46.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c003600 con 0x7f30100c3e30 2024-03-16T19:56:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c003780 con 0x7f30100c3e30 2024-03-16T19:56:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c003b70 con 0x7f30100c3e30 2024-03-16T19:56:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301014eed0 con 0x7f30100c3e30 2024-03-16T19:56:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301014f490 con 0x7f30100c3e30 2024-03-16T19:56:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.604+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3010068000 con 0x7f30100c3e30 2024-03-16T19:56:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.956+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f300c003cd0 con 0x7f30100c3e30 2024-03-16T19:56:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.956+0000 7f3006ffd640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 0x7f2ff4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.956+0000 7f3015d5d640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 0x7f2ff4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:46.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.956+0000 7f3015d5d640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 0x7f2ff4069350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30000ad000 tx=0x7f3000008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:46.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:46.964+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f300c055080 con 0x7f30100c3e30 2024-03-16T19:56:46.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.008+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f300c051320 con 0x7f30100c3e30 2024-03-16T19:56:47.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.188+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3010144880 con 0x7f30100c3e30 2024-03-16T19:56:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.452+0000 7f3006ffd640 1 -- 172.21.15.32:0/3325417498 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f300c017020 con 0x7f30100c3e30 2024-03-16T19:56:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:47.424 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.452+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 msgr2=0x7f2ff4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.452+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 0x7f2ff4069350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30000ad000 tx=0x7f3000008040 comp rx=0 tx=0).stop 2024-03-16T19:56:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.452+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 msgr2=0x7f3010145bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.452+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f300c00c490 tx=0x7f300c001e80 comp rx=0 tx=0).stop 2024-03-16T19:56:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 shutdown_connections 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2ff4066e90 0x7f2ff4069350 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010151220 0x7f301014aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30100c4210 0x7f3010143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 --2- 172.21.15.32:0/3325417498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30100c3e30 0x7f3010145bf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 >> 172.21.15.32:0/3325417498 conn(0x7f301006e6c0 msgr2=0x7f301006ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 shutdown_connections 2024-03-16T19:56:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.456+0000 7f30187c2640 1 -- 172.21.15.32:0/3325417498 wait complete. 2024-03-16T19:56:47.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:47.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.584+0000 7fc73309d640 1 Processor -- start 2024-03-16T19:56:47.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.584+0000 7fc73309d640 1 -- start start 2024-03-16T19:56:47.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73309d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc72c072670 0x7fc72c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73309d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c159f20 con 0x7fc72c0c3280 2024-03-16T19:56:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73309d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c156930 con 0x7fc72c150bb0 2024-03-16T19:56:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73309d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0c4350 con 0x7fc72c072670 2024-03-16T19:56:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73289c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc72c0c3280 0x7fc72c1612b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37534/0 (socket says 172.21.15.32:37534) 2024-03-16T19:56:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73289c640 1 -- 172.21.15.32:0/3288473070 learned_addr learned my addr 172.21.15.32:0/3288473070 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:47.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc73189a640 1 --2- 172.21.15.32:0/3288473070 >> v2:172.21.15.59:3300/0 conn(0x7fc72c072670 0x7fc72c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213106614 0 0) 0x7fc72c159f20 con 0x7fc72c0c3280 2024-03-16T19:56:47.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.588+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc718003530 con 0x7fc72c0c3280 2024-03-16T19:56:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229965982 0 0) 0x7fc718003530 con 0x7fc72c0c3280 2024-03-16T19:56:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 >> v2:172.21.15.59:3300/0 conn(0x7fc72c072670 msgr2=0x7fc72c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 --2- 172.21.15.32:0/3288473070 >> v2:172.21.15.59:3300/0 conn(0x7fc72c072670 0x7fc72c06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 >> v1:172.21.15.59:6789/0 conn(0x7fc72c150bb0 legacy=0x7fc72c1570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c0c3ad0 con 0x7fc72c0c3280 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.592+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fc724001a00 con 0x7fc72c0c3280 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 >> v1:172.21.15.32:6789/0 conn(0x7fc72c0c3280 legacy=0x7fc72c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc718003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7180047d0 0x7fc718004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc718005190 0x7fc71800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71800e220 con 0x7fc7180039d0 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71800e390 con 0x7fc718005190 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71800e550 con 0x7fc7180047d0 2024-03-16T19:56:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc73189a640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7180047d0 0x7fc718004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.596+0000 7fc73209b640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc718003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73209b640 1 -- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7180047d0 msgr2=0x7fc718004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73209b640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7180047d0 0x7fc718004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73209b640 1 -- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc718005190 msgr2=0x7fc71800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:47.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73209b640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc718005190 0x7fc71800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73209b640 1 -- 172.21.15.32:0/3288473070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718003530 con 0x7fc7180039d0 2024-03-16T19:56:47.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.600+0000 7fc73289c640 1 -- 172.21.15.32:0/3288473070 reap_dead start 2024-03-16T19:56:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.604+0000 7fc73209b640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc718003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc728003db0 tx=0x7fc72801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e3c467155b3ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:47.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280ab600 con 0x7fc7180039d0 2024-03-16T19:56:47.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7280a7050 con 0x7fc7180039d0 2024-03-16T19:56:47.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc731099640 1 -- 172.21.15.32:0/3288473070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280aba00 con 0x7fc7180039d0 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 msgr2=0x7fc718003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc718003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc728003db0 tx=0x7fc72801c320 comp rx=0 tx=0).stop 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- 172.21.15.32:0/3288473070 shutdown_connections 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 --2- 172.21.15.32:0/3288473070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc718003dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- 172.21.15.32:0/3288473070 >> 172.21.15.32:0/3288473070 conn(0x7fc72c0cc5d0 msgr2=0x7fc72c0cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- 172.21.15.32:0/3288473070 shutdown_connections 2024-03-16T19:56:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- 172.21.15.32:0/3288473070 wait complete. 2024-03-16T19:56:47.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 Processor -- start 2024-03-16T19:56:47.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- start start 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c0c7fd0 0x7fc72c155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc72c155ea0 0x7fc72c141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c159410 con 0x7fc7180039d0 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c154f00 con 0x7fc72c0c7fd0 2024-03-16T19:56:47.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.612+0000 7fc73309d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c158fd0 con 0x7fc72c155ea0 2024-03-16T19:56:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.616+0000 7fc73209b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.616+0000 7fc73209b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36394/0 (socket says 172.21.15.32:36394) 2024-03-16T19:56:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.616+0000 7fc73209b640 1 -- 172.21.15.32:0/162842433 learned_addr learned my addr 172.21.15.32:0/162842433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.616+0000 7fc73289c640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc72c155ea0 0x7fc72c141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.616+0000 7fc73189a640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c0c7fd0 0x7fc72c155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 -- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc72c155ea0 msgr2=0x7fc72c141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc72c155ea0 0x7fc72c141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 -- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c0c7fd0 msgr2=0x7fc72c155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:47.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c0c7fd0 0x7fc72c155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:47.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 -- 172.21.15.32:0/162842433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c158150 con 0x7fc7180039d0 2024-03-16T19:56:47.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.620+0000 7fc73209b640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc72801cbc0 tx=0x7fc718005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728007450 con 0x7fc7180039d0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7280075b0 con 0x7fc7180039d0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280b5630 con 0x7fc7180039d0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c1556c0 con 0x7fc7180039d0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c157b90 con 0x7fc7180039d0 2024-03-16T19:56:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.624+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c154d60 con 0x7fc7180039d0 2024-03-16T19:56:47.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.644+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc7280b2050 con 0x7fc7180039d0 2024-03-16T19:56:47.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.644+0000 7fc722ffd640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 0x7fc710069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:47.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.644+0000 7fc73189a640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 0x7fc710069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:47.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.648+0000 7fc73189a640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 0x7fc710069350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0c7f60 tx=0x7fc71c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:47.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.928+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc7280efc00 con 0x7fc7180039d0 2024-03-16T19:56:47.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:47.928+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7280ebea0 con 0x7fc7180039d0 2024-03-16T19:56:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.100+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc72c154d60 con 0x7fc7180039d0 2024-03-16T19:56:48.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.268+0000 7fc722ffd640 1 -- 172.21.15.32:0/162842433 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fc72c154d60 con 0x7fc7180039d0 2024-03-16T19:56:48.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:48.241 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 msgr2=0x7fc710069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 0x7fc710069350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0c7f60 tx=0x7fc71c009190 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 msgr2=0x7fc72c14c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc72801cbc0 tx=0x7fc718005800 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 shutdown_connections 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc710066e90 0x7fc710069350 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc72c155ea0 0x7fc72c141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c0c7fd0 0x7fc72c155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 --2- 172.21.15.32:0/162842433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7180039d0 0x7fc72c14c110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 >> 172.21.15.32:0/162842433 conn(0x7fc72c0cc5d0 msgr2=0x7fc72c06c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 shutdown_connections 2024-03-16T19:56:48.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.272+0000 7fc73309d640 1 -- 172.21.15.32:0/162842433 wait complete. 2024-03-16T19:56:48.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:56:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 Processor -- start 2024-03-16T19:56:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 -- start start 2024-03-16T19:56:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 0x7f4c940c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c940c3940 con 0x7f4c940c4210 2024-03-16T19:56:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94072cc0 con 0x7f4c940c3e30 2024-03-16T19:56:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c98fcc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c7ca0 con 0x7f4c94151220 2024-03-16T19:56:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c92d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4c940c4210 0x7f4c940c8630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37542/0 (socket says 172.21.15.32:37542) 2024-03-16T19:56:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c92d9d640 1 -- 172.21.15.32:0/2932465072 learned_addr learned my addr 172.21.15.32:0/2932465072 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.424+0000 7f4c91d9b640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 0x7f4c940c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.428+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128374188 0 0) 0x7f4c940c3940 con 0x7f4c940c4210 2024-03-16T19:56:48.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.428+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c7c005740 con 0x7f4c940c4210 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276637466 0 0) 0x7f4c7c005740 con 0x7f4c940c4210 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 msgr2=0x7f4c940c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 0x7f4c940c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 >> v1:172.21.15.59:6789/0 conn(0x7f4c940c3e30 legacy=0x7f4c94150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940c7780 con 0x7f4c940c4210 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f4c8c001a00 con 0x7f4c940c4210 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 >> v1:172.21.15.32:6789/0 conn(0x7f4c940c4210 legacy=0x7f4c940c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c005bc0 0x7f4c7c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 0x7f4c7c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c7c006f20 0x7f4c7c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c7ca0 con 0x7f4c7c006f20 2024-03-16T19:56:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c00dc50 con 0x7f4c7c005bc0 2024-03-16T19:56:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c00de10 con 0x7f4c94151220 2024-03-16T19:56:48.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.432+0000 7f4c92d9d640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c7c006f20 0x7f4c7c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.436+0000 7f4c92d9d640 1 -- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 msgr2=0x7f4c7c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.436+0000 7f4c92d9d640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c94151220 0x7f4c7c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.436+0000 7f4c92d9d640 1 -- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c005bc0 msgr2=0x7f4c7c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.436+0000 7f4c92d9d640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c005bc0 0x7f4c7c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.436+0000 7f4c92d9d640 1 -- 172.21.15.32:0/2932465072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c005740 con 0x7f4c7c006f20 2024-03-16T19:56:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.440+0000 7f4c92d9d640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c7c006f20 0x7f4c7c0073a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c002240 tx=0x7f4c8c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9163d3509bffa772 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:48.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c0072d0 con 0x7f4c7c006f20 2024-03-16T19:56:48.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c0a8070 con 0x7f4c7c006f20 2024-03-16T19:56:48.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c9159a640 1 -- 172.21.15.32:0/2932465072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c0076d0 con 0x7f4c7c006f20 2024-03-16T19:56:48.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 -- 172.21.15.32:0/2932465072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c7c006f20 msgr2=0x7f4c7c0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/2932465072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c7c006f20 0x7f4c7c0073a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c002240 tx=0x7f4c8c01d300 comp rx=0 tx=0).stop 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c92d9d640 1 -- 172.21.15.32:0/2932465072 reap_dead start 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 -- 172.21.15.32:0/2932465072 shutdown_connections 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 -- 172.21.15.32:0/2932465072 >> 172.21.15.32:0/2932465072 conn(0x7f4c9406e6c0 msgr2=0x7f4c9406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 -- 172.21.15.32:0/2932465072 shutdown_connections 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.444+0000 7f4c98fcc640 1 -- 172.21.15.32:0/2932465072 wait complete. 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 Processor -- start 2024-03-16T19:56:48.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 -- start start 2024-03-16T19:56:48.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c7c006f20 0x7f4c9414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c9414e430 0x7f4c94144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94150680 con 0x7f4c941567f0 2024-03-16T19:56:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94158fc0 con 0x7f4c9414e430 2024-03-16T19:56:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c98fcc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941eea20 con 0x7f4c7c006f20 2024-03-16T19:56:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c91d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c91d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36422/0 (socket says 172.21.15.32:36422) 2024-03-16T19:56:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c91d9b640 1 -- 172.21.15.32:0/617241228 learned_addr learned my addr 172.21.15.32:0/617241228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:48.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c9259c640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c7c006f20 0x7f4c9414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.448+0000 7f4c92d9d640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c9414e430 0x7f4c94144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 -- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c7c006f20 msgr2=0x7f4c9414f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c7c006f20 0x7f4c9414f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 -- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c9414e430 msgr2=0x7f4c94144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c9414e430 0x7f4c94144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 -- 172.21.15.32:0/617241228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c941ef2c0 con 0x7f4c941567f0 2024-03-16T19:56:48.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.452+0000 7f4c91d9b640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e7f0 tx=0x7f4c8800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.456+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8800c8c0 con 0x7f4c941567f0 2024-03-16T19:56:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.456+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8800ca20 con 0x7f4c941567f0 2024-03-16T19:56:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.456+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8800cdb0 con 0x7f4c941567f0 2024-03-16T19:56:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.456+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c941ebfd0 con 0x7f4c941567f0 2024-03-16T19:56:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.456+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c941ec240 con 0x7f4c941567f0 2024-03-16T19:56:48.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.460+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c941ef450 con 0x7f4c941567f0 2024-03-16T19:56:48.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.488+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4c88013470 con 0x7f4c941567f0 2024-03-16T19:56:48.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.492+0000 7f4c82ffd640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 0x7f4c74069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:48.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.492+0000 7f4c9259c640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 0x7f4c74069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:48.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.492+0000 7f4c9259c640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 0x7f4c74069350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c84007fd0 tx=0x7f4c84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.496+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4c88055df0 con 0x7f4c941567f0 2024-03-16T19:56:48.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:48.828+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4c88052090 con 0x7f4c941567f0 2024-03-16T19:56:49.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.032+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c94068000 con 0x7f4c941567f0 2024-03-16T19:56:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.064+0000 7f4c82ffd640 1 -- 172.21.15.32:0/617241228 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6550 (secure 0 0 0) 0x7f4c88017090 con 0x7f4c941567f0 2024-03-16T19:56:49.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:49.036 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:56:44.497940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4255,"qdb_cluster":[4255]},"id":1}]} 2024-03-16T19:56:49.037 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-03-16T19:56:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 msgr2=0x7f4c74069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 0x7f4c74069350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c84007fd0 tx=0x7f4c84009040 comp rx=0 tx=0).stop 2024-03-16T19:56:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 msgr2=0x7f4c94144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e7f0 tx=0x7f4c8800ecb0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 shutdown_connections 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c74066e90 0x7f4c74069350 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c9414e430 0x7f4c94144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c941567f0 0x7f4c94144a40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 --2- 172.21.15.32:0/617241228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c7c006f20 0x7f4c9414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 >> 172.21.15.32:0/617241228 conn(0x7f4c9406e6c0 msgr2=0x7f4c940c5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 shutdown_connections 2024-03-16T19:56:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.068+0000 7f4c98fcc640 1 -- 172.21.15.32:0/617241228 wait complete. 2024-03-16T19:56:49.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:49.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 Processor -- start 2024-03-16T19:56:49.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 -- start start 2024-03-16T19:56:49.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d1c067140 0x7f9d1c072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1c157d40 con 0x7f9d1c151fb0 2024-03-16T19:56:49.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1c0c6c60 con 0x7f9d1c066cd0 2024-03-16T19:56:49.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d23dff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c0c6f40 con 0x7f9d1c067140 2024-03-16T19:56:49.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d21b9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9d1c151fb0 0x7f9d1c06afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58438/0 (socket says 172.21.15.32:58438) 2024-03-16T19:56:49.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.200+0000 7f9d21b9b640 1 -- 172.21.15.32:0/94155487 learned_addr learned my addr 172.21.15.32:0/94155487 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.204+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663364159 0 0) 0x7f9d1c157d40 con 0x7f9d1c151fb0 2024-03-16T19:56:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.204+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d08003530 con 0x7f9d1c151fb0 2024-03-16T19:56:49.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132948093 0 0) 0x7f9d08003530 con 0x7f9d1c151fb0 2024-03-16T19:56:49.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d1c067140 msgr2=0x7f9d1c072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d1c067140 0x7f9d1c072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 >> v1:172.21.15.59:6789/0 conn(0x7f9d1c066cd0 legacy=0x7f9d1c1625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c158000 con 0x7f9d1c151fb0 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f9d18001680 con 0x7f9d1c151fb0 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 >> v1:172.21.15.32:6789/0 conn(0x7f9d1c151fb0 legacy=0x7f9d1c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d080039b0 0x7f9d08003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080047d0 0x7f9d08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080052b0 0x7f9d0800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c0c6f40 con 0x7f9d080047d0 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0800c060 con 0x7f9d080039b0 2024-03-16T19:56:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.208+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0800c220 con 0x7f9d080052b0 2024-03-16T19:56:49.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.212+0000 7f9d2139a640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080047d0 0x7f9d08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.216+0000 7f9d2139a640 1 -- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080052b0 msgr2=0x7f9d0800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.216+0000 7f9d2139a640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080052b0 0x7f9d0800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.216+0000 7f9d2139a640 1 -- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d080039b0 msgr2=0x7f9d08003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.216+0000 7f9d2139a640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d080039b0 0x7f9d08003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.216+0000 7f9d2139a640 1 -- 172.21.15.32:0/94155487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d08003530 con 0x7f9d080047d0 2024-03-16T19:56:49.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.220+0000 7f9d2139a640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080047d0 0x7f9d08004c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009890 tx=0x7f9d0c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c63275c11eba30 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0ab600 con 0x7f9d080047d0 2024-03-16T19:56:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0c0a7050 con 0x7f9d080047d0 2024-03-16T19:56:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d20b99640 1 -- 172.21.15.32:0/94155487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0aba00 con 0x7f9d080047d0 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 -- 172.21.15.32:0/94155487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080047d0 msgr2=0x7f9d08004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 --2- 172.21.15.32:0/94155487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080047d0 0x7f9d08004c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009890 tx=0x7f9d0c01c3c0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d2239c640 1 -- 172.21.15.32:0/94155487 reap_dead start 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 -- 172.21.15.32:0/94155487 shutdown_connections 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 -- 172.21.15.32:0/94155487 >> 172.21.15.32:0/94155487 conn(0x7f9d1c06f560 msgr2=0x7f9d1c06f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 -- 172.21.15.32:0/94155487 shutdown_connections 2024-03-16T19:56:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.224+0000 7f9d23dff640 1 -- 172.21.15.32:0/94155487 wait complete. 2024-03-16T19:56:49.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 Processor -- start 2024-03-16T19:56:49.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 -- start start 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080047d0 0x7f9d1c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d1c1ef2e0 0x7f9d1c1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1f0310 con 0x7f9d1c1561a0 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1570b0 con 0x7f9d1c1ef2e0 2024-03-16T19:56:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d23dff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1ef790 con 0x7f9d080047d0 2024-03-16T19:56:49.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d2239c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:49.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d2239c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39700/0 (socket says 172.21.15.32:39700) 2024-03-16T19:56:49.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.228+0000 7f9d2239c640 1 -- 172.21.15.32:0/2771579697 learned_addr learned my addr 172.21.15.32:0/2771579697 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.232+0000 7f9d2239c640 1 -- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080047d0 msgr2=0x7f9d1c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.232+0000 7f9d2239c640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080047d0 0x7f9d1c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.232+0000 7f9d2239c640 1 -- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d1c1ef2e0 msgr2=0x7f9d1c1f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.232+0000 7f9d2239c640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d1c1ef2e0 0x7f9d1c1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.232+0000 7f9d2239c640 1 -- 172.21.15.32:0/2771579697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c1efa70 con 0x7f9d1c1561a0 2024-03-16T19:56:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.236+0000 7f9d2239c640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004750 tx=0x7f9d1400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14007110 con 0x7f9d1c1561a0 2024-03-16T19:56:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d14007270 con 0x7f9d1c1561a0 2024-03-16T19:56:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c1efd50 con 0x7f9d1c1561a0 2024-03-16T19:56:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14007620 con 0x7f9d1c1561a0 2024-03-16T19:56:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c1f2a70 con 0x7f9d1c1561a0 2024-03-16T19:56:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.240+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c1f28d0 con 0x7f9d1c1561a0 2024-03-16T19:56:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.532+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9d14007780 con 0x7f9d1c1561a0 2024-03-16T19:56:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.536+0000 7f9d127fc640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 0x7f9d00069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.536+0000 7f9d21b9b640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 0x7f9d00069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.536+0000 7f9d21b9b640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 0x7f9d00069380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d180079c0 tx=0x7f9d180015d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:49.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.548+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9d1401b2f0 con 0x7f9d1c1561a0 2024-03-16T19:56:49.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.548+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9d14051560 con 0x7f9d1c1561a0 2024-03-16T19:56:49.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.728+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d1c1f28d0 con 0x7f9d1c1561a0 2024-03-16T19:56:49.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.780+0000 7f9d127fc640 1 -- 172.21.15.32:0/2771579697 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f9d1c1f28d0 con 0x7f9d1c1561a0 2024-03-16T19:56:49.753 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:49.753 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 msgr2=0x7f9d00069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 0x7f9d00069380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d180079c0 tx=0x7f9d180015d0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 msgr2=0x7f9d1c1f34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004750 tx=0x7f9d1400c980 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 shutdown_connections 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d00066ec0 0x7f9d00069380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d1c1561a0 0x7f9d1c1f34d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d1c1ef2e0 0x7f9d1c1f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 --2- 172.21.15.32:0/2771579697 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d080047d0 0x7f9d1c0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 >> 172.21.15.32:0/2771579697 conn(0x7f9d1c06f560 msgr2=0x7f9d1c14c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 shutdown_connections 2024-03-16T19:56:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:49.784+0000 7f9d23dff640 1 -- 172.21.15.32:0/2771579697 wait complete. 2024-03-16T19:56:50.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:56:44.497940+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4255, 'qdb_cluster': [4255]} 2024-03-16T19:56:50.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-16T19:56:50.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-16T19:56:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 Processor -- start 2024-03-16T19:56:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 -- start start 2024-03-16T19:56:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08e00c8ef0 0x7f08e0150d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00c2dd0 con 0x7f08e0071c90 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e00c3070 con 0x7f08e00c8ef0 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.520+0000 7f08e63c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00c3350 con 0x7f08e00718b0 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08e00c8ef0 0x7f08e0150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08e00c8ef0 0x7f08e0150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39712/0 (socket says 172.21.15.32:39712) 2024-03-16T19:56:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 -- 172.21.15.32:0/1032652175 learned_addr learned my addr 172.21.15.32:0/1032652175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 -- 172.21.15.32:0/1032652175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 msgr2=0x7f08e00c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 --2- 172.21.15.32:0/1032652175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 -- 172.21.15.32:0/1032652175 >> v1:172.21.15.59:6789/0 conn(0x7f08e0071c90 legacy=0x7f08e0072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.524+0000 7f08e4962640 1 -- 172.21.15.32:0/1032652175 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e00c3630 con 0x7f08e00c8ef0 2024-03-16T19:56:50.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.528+0000 7f08e4962640 1 --2- 172.21.15.32:0/1032652175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e00c8ef0 0x7f08e0150d10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009e50 tx=0x7f08d801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e36171da978d261 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.532+0000 7f08deffd640 1 -- 172.21.15.32:0/1032652175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80af400 con 0x7f08e00c8ef0 2024-03-16T19:56:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.532+0000 7f08deffd640 1 -- 172.21.15.32:0/1032652175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d80ab050 con 0x7f08e00c8ef0 2024-03-16T19:56:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.532+0000 7f08deffd640 1 -- 172.21.15.32:0/1032652175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80af780 con 0x7f08e00c8ef0 2024-03-16T19:56:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.532+0000 7f08e63c5640 1 -- 172.21.15.32:0/1032652175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e00c8ef0 msgr2=0x7f08e0150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.532+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1032652175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e00c8ef0 0x7f08e0150d10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009e50 tx=0x7f08d801f210 comp rx=0 tx=0).stop 2024-03-16T19:56:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- 172.21.15.32:0/1032652175 shutdown_connections 2024-03-16T19:56:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1032652175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e00c8ef0 0x7f08e0150d10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1032652175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e00c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- 172.21.15.32:0/1032652175 >> 172.21.15.32:0/1032652175 conn(0x7f08e00c83d0 msgr2=0x7f08e00c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- 172.21.15.32:0/1032652175 shutdown_connections 2024-03-16T19:56:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- 172.21.15.32:0/1032652175 wait complete. 2024-03-16T19:56:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 Processor -- start 2024-03-16T19:56:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- start start 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e01f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08e00c8ef0 0x7f08e00c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e01f3990 con 0x7f08e0071c90 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00cad30 con 0x7f08e00c8ef0 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e63c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00cb040 con 0x7f08e00718b0 2024-03-16T19:56:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39724/0 (socket says 172.21.15.32:39724) 2024-03-16T19:56:50.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08df7fe640 1 -- 172.21.15.32:0/1922140016 learned_addr learned my addr 172.21.15.32:0/1922140016 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.536+0000 7f08e4962640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08e00c8ef0 0x7f08e00c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.540+0000 7f08df7fe640 1 -- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 msgr2=0x7f08e01f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.540+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e01f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.540+0000 7f08df7fe640 1 -- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08e00c8ef0 msgr2=0x7f08e00c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.540+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08e00c8ef0 0x7f08e00c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.540+0000 7f08df7fe640 1 -- 172.21.15.32:0/1922140016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8009b30 con 0x7f08e0071c90 2024-03-16T19:56:50.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.544+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c430 tx=0x7f08d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08dcff9640 1 -- 172.21.15.32:0/1922140016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40052d0 con 0x7f08e0071c90 2024-03-16T19:56:50.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08dcff9640 1 -- 172.21.15.32:0/1922140016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4005430 con 0x7f08e0071c90 2024-03-16T19:56:50.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e00cb320 con 0x7f08e0071c90 2024-03-16T19:56:50.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08dcff9640 1 -- 172.21.15.32:0/1922140016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4007260 con 0x7f08e0071c90 2024-03-16T19:56:50.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e00ca6e0 con 0x7f08e0071c90 2024-03-16T19:56:50.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 Processor -- start 2024-03-16T19:56:50.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 -- start start 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c19350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c54680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620adbd00 con 0x561620c19890 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620a6d430 con 0x561620c54bc0 2024-03-16T19:56:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.548+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x561620a6a960 con 0x561620c52f70 2024-03-16T19:56:50.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.552+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.552+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:38172/0 (socket says 172.21.15.32:38172) 2024-03-16T19:56:50.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.552+0000 7f08df7fe640 1 -- 172.21.15.32:0/1169273870 learned_addr learned my addr 172.21.15.32:0/1169273870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.556+0000 7f08df7fe640 1 -- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 msgr2=0x561620c19350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.556+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c19350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.556+0000 7f08df7fe640 1 -- 172.21.15.32:0/1169273870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 msgr2=0x561620c54680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.556+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c54680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.556+0000 7f08df7fe640 1 -- 172.21.15.32:0/1169273870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56162075c890 con 0x561620c54bc0 2024-03-16T19:56:50.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.560+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08d4004000 tx=0x7f08d4011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10688e553f0f37b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08dcff9640 1 -- 172.21.15.32:0/1922140016 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08d4003ad0 con 0x7f08e0071c90 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08c5ffb640 1 -- 172.21.15.32:0/1169273870 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4085670 con 0x561620c54bc0 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08c5ffb640 1 -- 172.21.15.32:0/1169273870 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4085830 con 0x561620c54bc0 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08c5ffb640 1 -- 172.21.15.32:0/1169273870 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4085ca0 con 0x561620c54bc0 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08dcff9640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 0x7f08cc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e4962640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 0x7f08cc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 msgr2=0x561620c54fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08d4004000 tx=0x7f08d4011000 comp rx=0 tx=0).stop 2024-03-16T19:56:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- 172.21.15.32:0/1169273870 shutdown_connections 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c54fa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c54680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 --2- 172.21.15.32:0/1169273870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c19350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- 172.21.15.32:0/1169273870 >> 172.21.15.32:0/1169273870 conn(0x561620c19ff0 msgr2=0x561620c52440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- 172.21.15.32:0/1169273870 shutdown_connections 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e4962640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 0x7f08cc069350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f08d80b1000 tx=0x7f08d8004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- 172.21.15.32:0/1169273870 wait complete. 2024-03-16T19:56:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 -- start start 2024-03-16T19:56:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.568+0000 7f08e9537000 1 Processor -- start 2024-03-16T19:56:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c5fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620a66140 con 0x561620c19890 2024-03-16T19:56:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620a00630 con 0x561620c54bc0 2024-03-16T19:56:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x561620a002d0 con 0x561620c52f70 2024-03-16T19:56:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e9537000 10 client.0 _ll_register_callbacks cb 0x561620b161c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T19:56:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08dcff9640 1 -- 172.21.15.32:0/1922140016 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f08d40073c0 con 0x7f08e0071c90 2024-03-16T19:56:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.572+0000 7f08e4962640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.576+0000 7f08e4962640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:38178/0 (socket says 172.21.15.32:38178) 2024-03-16T19:56:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.576+0000 7f08e4962640 1 -- 172.21.15.32:0/3798394418 learned_addr learned my addr 172.21.15.32:0/3798394418 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.576+0000 7f08dffff640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c5fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.580+0000 7f08e4962640 1 -- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 msgr2=0x561620c602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.580+0000 7f08e4962640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.580+0000 7f08e4962640 1 -- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 msgr2=0x561620c5fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.580+0000 7f08e4962640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c5fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.580+0000 7f08e4962640 1 -- 172.21.15.32:0/3798394418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e00ca6e0 con 0x561620c54bc0 2024-03-16T19:56:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.584+0000 7f08e4962640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08d801fab0 tx=0x7f08d801da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7e3ff7d93a880b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.584+0000 7f08beffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-16T19:56:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.592+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80bab40 con 0x561620c54bc0 2024-03-16T19:56:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.592+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f08d80baca0 con 0x561620c54bc0 2024-03-16T19:56:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.592+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561620a19850 con 0x561620c54bc0 2024-03-16T19:56:50.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.600+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08d80afe90 con 0x561620c54bc0 2024-03-16T19:56:50.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.600+0000 7f08e9537000 10 client.4330 fetch_fsmap learned FSMap version 7 2024-03-16T19:56:50.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.600+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x561620a25e70 con 0x561620c54bc0 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561620a19850 con 0x561620c54bc0 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 7) v1 ==== 4472+0+0 (secure 0 0 0) 0x7f08d80c2040 con 0x561620c54bc0 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 10 client.4330 fetch_fsmap finished waiting for FSMap version 7 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 10 client.4330 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 20 client.4330 populate_metadata read hostname 'smithi032' 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c5e140 0x561620c5e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 4 client.4330 mds_command: new command op to 4255 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T19:56:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.632+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 --> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561620a62f20 con 0x561620c5e140 2024-03-16T19:56:50.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.648+0000 7f08dffff640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c5e140 0x561620c5e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.744+0000 7f08dffff640 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c5e140 0x561620c5e520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c9a54079dd8710 server_cookie=afe8d1507c6b7466 in_seq=0 out_seq=0 2024-03-16T19:56:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.744+0000 7f08beffd640 10 client.4330 ms_handle_connect on v2:172.21.15.32:6838/4212779160 2024-03-16T19:56:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08beffd640 1 -- 172.21.15.32:0/3798394418 <== mds.0 v2:172.21.15.32:6838/4212779160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561620a62f20 con 0x561620c5e140 2024-03-16T19:56:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08beffd640 10 client.4330 handle_command_reply: tid=0 2024-03-16T19:56:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 client.4330 shutdown 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 msgr2=0x561620c4edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08d801fab0 tx=0x7f08d801da50 comp rx=0 tx=0).stop 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 shutdown_connections 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c54bc0 0x561620c4edf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c5e140 0x561620c5e520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 --2- 172.21.15.32:0/3798394418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c5fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 >> 172.21.15.32:0/3798394418 conn(0x561620c19ff0 msgr2=0x561620c52020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.920+0000 7f08beffd640 0 client.4330 ms_handle_reset on v2:172.21.15.32:6838/4212779160 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.924+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 shutdown_connections 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.924+0000 7f08e9537000 1 -- 172.21.15.32:0/3798394418 wait complete. 2024-03-16T19:56:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.924+0000 7f08e9537000 20 client.4330 trim_cache size 0 max 16384 2024-03-16T19:56:50.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.948+0000 7f08e9537000 1 Processor -- start 2024-03-16T19:56:50.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.948+0000 7f08e9537000 1 -- start start 2024-03-16T19:56:50.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c19ff0 0x561620c53a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c66b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620c18b50 con 0x561620c19890 2024-03-16T19:56:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162071d940 con 0x561620c19ff0 2024-03-16T19:56:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x561620c17ca0 con 0x561620c52f70 2024-03-16T19:56:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39746/0 (socket says 172.21.15.32:39746) 2024-03-16T19:56:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.952+0000 7f08df7fe640 1 -- 172.21.15.32:0/4181194826 learned_addr learned my addr 172.21.15.32:0/4181194826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.956+0000 7f08df7fe640 1 -- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 msgr2=0x561620c66b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.956+0000 7f08df7fe640 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c66b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.956+0000 7f08df7fe640 1 -- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c19ff0 msgr2=0x561620c53a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.956+0000 7f08df7fe640 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c19ff0 0x561620c53a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.956+0000 7f08df7fe640 1 -- 172.21.15.32:0/4181194826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561620a0be10 con 0x561620c19890 2024-03-16T19:56:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.960+0000 7f08df7fe640 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08d40050a0 tx=0x7f08d4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc802f3e553e163c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08c57fa640 1 -- 172.21.15.32:0/4181194826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4085650 con 0x561620c19890 2024-03-16T19:56:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08c57fa640 1 -- 172.21.15.32:0/4181194826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4057de0 con 0x561620c19890 2024-03-16T19:56:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08c57fa640 1 -- 172.21.15.32:0/4181194826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40576e0 con 0x561620c19890 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 -- 172.21.15.32:0/4181194826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 msgr2=0x561620c665e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08d40050a0 tx=0x7f08d4000bd0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 -- 172.21.15.32:0/4181194826 shutdown_connections 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c52f70 0x561620c66b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19890 0x561620c665e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 --2- 172.21.15.32:0/4181194826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c19ff0 0x561620c53a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 -- 172.21.15.32:0/4181194826 >> 172.21.15.32:0/4181194826 conn(0x561620c185d0 msgr2=0x561620c53350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 -- 172.21.15.32:0/4181194826 shutdown_connections 2024-03-16T19:56:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.964+0000 7f08e9537000 1 -- 172.21.15.32:0/4181194826 wait complete. 2024-03-16T19:56:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 -- start start 2024-03-16T19:56:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 Processor -- start 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c19890 0x561620c65aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c52f70 0x561620cc8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620a6de80 con 0x561620c19ff0 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x561620a68ca0 con 0x561620c52f70 2024-03-16T19:56:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x561620a63950 con 0x561620c19890 2024-03-16T19:56:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08e9537000 10 client.0 _ll_register_callbacks cb 0x7f08d8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T19:56:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08df7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39760/0 (socket says 172.21.15.32:39760) 2024-03-16T19:56:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.968+0000 7f08df7fe640 1 -- 172.21.15.32:0/1320902837 learned_addr learned my addr 172.21.15.32:0/1320902837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 -- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c19890 msgr2=0x561620c65aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c19890 0x561620c65aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 -- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c52f70 msgr2=0x561620cc8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c52f70 0x561620cc8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 -- 172.21.15.32:0/1320902837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56162075c890 con 0x561620c19ff0 2024-03-16T19:56:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08df7fe640 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08d402b000 tx=0x7f08d4000f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ea72bff080d456 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.976+0000 7f08beffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-16T19:56:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.984+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d405b830 con 0x561620c19ff0 2024-03-16T19:56:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.984+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f08d4084030 con 0x561620c19ff0 2024-03-16T19:56:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.984+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616209ebe80 con 0x561620c19ff0 2024-03-16T19:56:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.996+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08d405fdf0 con 0x561620c19ff0 2024-03-16T19:56:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.996+0000 7f08e9537000 10 client.4374 fetch_fsmap learned FSMap version 7 2024-03-16T19:56:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:50.996+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x56162075ccc0 con 0x561620c19ff0 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5616209ebe80 con 0x561620c19ff0 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 7) v1 ==== 4472+0+0 (secure 0 0 0) 0x7f08d4053d50 con 0x561620c19ff0 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 10 client.4374 fetch_fsmap finished waiting for FSMap version 7 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 10 client.4374 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 20 client.4374 populate_metadata read hostname 'smithi032' 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c54bc0 0x561620cc88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 4 client.4374 mds_command: new command op to 4255 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T19:56:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.000+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 --> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561620a2efa0 con 0x561620c54bc0 2024-03-16T19:56:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.004+0000 7f08dffff640 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c54bc0 0x561620cc88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:50.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.008+0000 7f08beffd640 10 client.4374 ms_handle_connect on v2:172.21.15.32:6838/4212779160 2024-03-16T19:56:50.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.008+0000 7f08dffff640 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c54bc0 0x561620cc88d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f6824755ec933f server_cookie=56116e314c6e8db1 in_seq=0 out_seq=0 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08beffd640 1 -- 172.21.15.32:0/1320902837 <== mds.0 v2:172.21.15.32:6838/4212779160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x561620a2efa0 con 0x561620c54bc0 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08beffd640 10 client.4374 handle_command_reply: tid=0 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 client.4374 shutdown 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 msgr2=0x561620cc7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08d402b000 tx=0x7f08d4000f80 comp rx=0 tx=0).stop 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 shutdown_connections 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x561620c52f70 0x561620cc8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:6838/4212779160,v1:172.21.15.32:6839/4212779160] conn(0x561620c54bc0 0x561620cc88d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561620c19ff0 0x561620cc7e50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 --2- 172.21.15.32:0/1320902837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x561620c19890 0x561620c65aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08beffd640 0 client.4374 ms_handle_reset on v2:172.21.15.32:6838/4212779160 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 >> 172.21.15.32:0/1320902837 conn(0x561620c185d0 msgr2=0x561620c53be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 shutdown_connections 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 1 -- 172.21.15.32:0/1320902837 wait complete. 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e9537000 20 client.4374 trim_cache size 0 max 16384 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "fb85817b-8df0-4e27-a0bc-dc277b22543a", 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4255, 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-16T19:56:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 13.734721428, 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 7, 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 29, 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 29, 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 31.738374487000002 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T19:56:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 msgr2=0x7f08cc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 0x7f08cc069350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f08d80b1000 tx=0x7f08d8004ac0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 msgr2=0x7f08e00c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c430 tx=0x7f08d400c8f0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 shutdown_connections 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08cc066e90 0x7f08cc069350 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08e00c8ef0 0x7f08e00c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08e0071c90 0x7f08e00c6260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 --2- 172.21.15.32:0/1922140016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08e00718b0 0x7f08e01f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 >> 172.21.15.32:0/1922140016 conn(0x7f08e00c83d0 msgr2=0x7f08e00c4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 shutdown_connections 2024-03-16T19:56:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.028+0000 7f08e63c5640 1 -- 172.21.15.32:0/1922140016 wait complete. 2024-03-16T19:56:51.019 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T19:56:51.019 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:51.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-16T19:56:51.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 Processor -- start 2024-03-16T19:56:51.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 -- start start 2024-03-16T19:56:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f494c072d50 0x7f494c15b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 0x7f494c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0c7c90 con 0x7f494c151ef0 2024-03-16T19:56:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0c70d0 con 0x7f494c072d50 2024-03-16T19:56:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4950fc1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c072a00 con 0x7f494c0c7800 2024-03-16T19:56:51.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f494ad9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 0x7f494c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f4949d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f494c072d50 0x7f494c15b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f494a59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f494c151ef0 0x7f494c0722f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58450/0 (socket says 172.21.15.32:58450) 2024-03-16T19:56:51.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.172+0000 7f494a59c640 1 -- 172.21.15.32:0/1103454068 learned_addr learned my addr 172.21.15.32:0/1103454068 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.176+0000 7f494959a640 1 -- 172.21.15.32:0/1103454068 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700242009 0 0) 0x7f494c0c7c90 con 0x7f494c151ef0 2024-03-16T19:56:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.176+0000 7f494959a640 1 -- 172.21.15.32:0/1103454068 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934003530 con 0x7f494c151ef0 2024-03-16T19:56:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.180+0000 7f494ad9d640 1 -- 172.21.15.32:0/1103454068 >> v2:172.21.15.59:3300/0 conn(0x7f494c072d50 msgr2=0x7f494c15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.180+0000 7f494ad9d640 1 --2- 172.21.15.32:0/1103454068 >> v2:172.21.15.59:3300/0 conn(0x7f494c072d50 0x7f494c15b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.180+0000 7f494ad9d640 1 -- 172.21.15.32:0/1103454068 >> v1:172.21.15.32:6789/0 conn(0x7f494c151ef0 legacy=0x7f494c0722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.180+0000 7f494ad9d640 1 -- 172.21.15.32:0/1103454068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0c96b0 con 0x7f494c0c7800 2024-03-16T19:56:51.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.184+0000 7f494ad9d640 1 --2- 172.21.15.32:0/1103454068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 0x7f494c0c5de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4944009d90 tx=0x7f494401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c929c9d08865dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f494959a640 1 -- 172.21.15.32:0/1103454068 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440af600 con 0x7f494c0c7800 2024-03-16T19:56:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f494959a640 1 -- 172.21.15.32:0/1103454068 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440ab050 con 0x7f494c0c7800 2024-03-16T19:56:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f494959a640 1 -- 172.21.15.32:0/1103454068 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440af980 con 0x7f494c0c7800 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- 172.21.15.32:0/1103454068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 msgr2=0x7f494c0c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- 172.21.15.32:0/1103454068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 0x7f494c0c5de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4944009d90 tx=0x7f494401f150 comp rx=0 tx=0).stop 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- 172.21.15.32:0/1103454068 shutdown_connections 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- 172.21.15.32:0/1103454068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c0c7800 0x7f494c0c5de0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- 172.21.15.32:0/1103454068 >> v2:172.21.15.59:3300/0 conn(0x7f494c072d50 0x7f494c15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- 172.21.15.32:0/1103454068 >> 172.21.15.32:0/1103454068 conn(0x7f494c06e6c0 msgr2=0x7f494c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- 172.21.15.32:0/1103454068 shutdown_connections 2024-03-16T19:56:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- 172.21.15.32:0/1103454068 wait complete. 2024-03-16T19:56:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 Processor -- start 2024-03-16T19:56:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- start start 2024-03-16T19:56:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c072d50 0x7f494c1f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c151ef0 0x7f494c1ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c142180 con 0x7f494c0c7800 2024-03-16T19:56:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c141ee0 con 0x7f494c072d50 2024-03-16T19:56:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.196+0000 7f4950fc1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c145cf0 con 0x7f494c151ef0 2024-03-16T19:56:51.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.200+0000 7f4949d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.200+0000 7f4949d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39768/0 (socket says 172.21.15.32:39768) 2024-03-16T19:56:51.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.200+0000 7f4949d9b640 1 -- 172.21.15.32:0/183824 learned_addr learned my addr 172.21.15.32:0/183824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:51.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.200+0000 7f494a59c640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c072d50 0x7f494c1f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.200+0000 7f494ad9d640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c151ef0 0x7f494c1ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 -- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c151ef0 msgr2=0x7f494c1ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c151ef0 0x7f494c1ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 -- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c072d50 msgr2=0x7f494c1f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c072d50 0x7f494c1f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 -- 172.21.15.32:0/183824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944009a70 con 0x7f494c0c7800 2024-03-16T19:56:51.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.204+0000 7f4949d9b640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e930 tx=0x7f493c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:51.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009690 con 0x7f494c0c7800 2024-03-16T19:56:51.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c00ca00 con 0x7f494c0c7800 2024-03-16T19:56:51.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c00cd70 con 0x7f494c0c7800 2024-03-16T19:56:51.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c0cc980 con 0x7f494c0c7800 2024-03-16T19:56:51.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c15bff0 con 0x7f494c0c7800 2024-03-16T19:56:51.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.208+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c1ee800 con 0x7f494c0c7800 2024-03-16T19:56:51.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.228+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f493c013470 con 0x7f494c0c7800 2024-03-16T19:56:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.232+0000 7f493affd640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 0x7f492c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.232+0000 7f494a59c640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 0x7f492c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:51.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.232+0000 7f494a59c640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 0x7f492c069380 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4940007fd0 tx=0x7f4940009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.516+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f493c055eb0 con 0x7f494c0c7800 2024-03-16T19:56:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.516+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f493c052150 con 0x7f494c0c7800 2024-03-16T19:56:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.728+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f494c1ee800 con 0x7f494c0c7800 2024-03-16T19:56:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.736+0000 7f493affd640 1 -- 172.21.15.32:0/183824 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f493c021070 con 0x7f494c0c7800 2024-03-16T19:56:51.709 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-16T19:56:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 msgr2=0x7f492c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 0x7f492c069380 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4940007fd0 tx=0x7f4940009040 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 msgr2=0x7f494c1eeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e930 tx=0x7f493c00edf0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 shutdown_connections 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f492c066ec0 0x7f492c069380 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f494c151ef0 0x7f494c1ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c0c7800 0x7f494c1eeae0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 --2- 172.21.15.32:0/183824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c072d50 0x7f494c1f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 >> 172.21.15.32:0/183824 conn(0x7f494c06e6c0 msgr2=0x7f494c0c5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 shutdown_connections 2024-03-16T19:56:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:51.740+0000 7f4950fc1640 1 -- 172.21.15.32:0/183824 wait complete. 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T19:56:51.729 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T19:56:51.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:51.730 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-16T19:56:51.735 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T19:56:51.735 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T19:56:51.735 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-16T19:56:51.736 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42143sec preferred_lft 42143sec 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 02:42:33:25:86:02 brd ff:ff:ff:ff:ff:ff 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-16T19:56:51.737 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:56:51.738 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T19:56:51.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:51.738 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-16T19:56:51.738 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-16T19:56:51.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-03-16T19:56:51.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-03-16T19:56:51.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-03-16T19:56:51.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T19:56:51.739 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-16T19:56:51.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:51.816 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T19:56:51.866 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-16T19:56:51.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:51.867 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-16T19:56:51.932 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T19:56:51.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-16T19:56:51.933 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-16T19:56:52.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.000 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-16T19:56:52.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.050 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-03-16T19:56:52.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.102 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-16T19:56:52.102 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-16T19:56:52.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:52.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-16T19:56:52.102 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-16T19:56:52.171 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-16T19:56:52.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.171 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-16T19:56:52.172 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-16T19:56:52.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.273 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-16T19:56:52.273 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-16T19:56:52.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-16T19:56:52.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T19:56:52.273 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-16T19:56:52.290 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-16T19:56:52.290 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T19:56:52.290 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T19:56:52.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T19:56:52.338 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:52.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-03-16T19:56:52.441 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/run/lock 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-03-16T19:56:52.487 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/fs/nfsd 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysusers.service 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/run/rpc_pipefs 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-0 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-1 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-2 2024-03-16T19:56:52.488 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-3 2024-03-16T19:56:52.489 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-03-16T19:56:52.489 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:52.489 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:52.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.489 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-03-16T19:56:52.533 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-16T19:56:52.534 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-16T19:56:52.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-03-16T19:56:52.591 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T19:56:52.638 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-03-16T19:56:52.638 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-03-16T19:56:52.638 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-03-16T19:56:52.638 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-03-16T19:56:52.638 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/run/lock 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/fs/nfsd 2024-03-16T19:56:52.639 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysusers.service 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/rpc_pipefs 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-0 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-1 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-2 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-3 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:52.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:52.641 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-03-16T19:56:52.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:2024-03-16T19:56:52.708+0000 7f0144ef4580 -1 init, newargv = 0x562512ece1c0 newargc=15 2024-03-16T19:56:52.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[32742]: starting ceph client 2024-03-16T19:56:53.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[32742]: starting fuse 2024-03-16T19:56:53.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-03-16T19:56:53.800 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T19:56:53.848 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/run/lock 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-03-16T19:56:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/fs/nfsd 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysusers.service 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/run/rpc_pipefs 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-0 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-1 2024-03-16T19:56:53.850 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-2 2024-03-16T19:56:53.851 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph/osd/ceph-3 2024-03-16T19:56:53.851 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-03-16T19:56:53.851 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:53.851 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T19:56:53.851 INFO:teuthology.orchestra.run.smithi032.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:53.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:53.852 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-03-16T19:56:53.898 INFO:teuthology.orchestra.run.smithi032.stdout:45 2024-03-16T19:56:53.898 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo stdin-killer -- python3 -c ' 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> import re 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> def _find_admin_socket(client_name): 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2024-03-16T19:56:53.899 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> contents = proc_f.read() 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> if mountpoint in contents: 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> return f 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> print(_find_admin_socket("client.admin")) 2024-03-16T19:56:53.900 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-16T19:56:54.024 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.admin.32742.asok 2024-03-16T19:56:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T19:56:54.036 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.32742.asok 2024-03-16T19:56:54.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:54.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.32742.asok status 2024-03-16T19:56:54.200 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev)", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "admin", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "32742", 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2024-03-16T19:56:54.201 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4339, 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 4339 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3632831408 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3632831408 2024-03-16T19:56:54.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.4339 172.21.15.32:0/3632831408", 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3632831408", 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 7, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 29, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklisted": false, 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2024-03-16T19:56:54.203 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T19:56:54.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:54.215 DEBUG:teuthology.orchestra.run.smithi032:> stat --file-system '--printf=%T 2024-03-16T19:56:54.215 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:54.255 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2024-03-16T19:56:54.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:54.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:54.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:56:54.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:56:54.393 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-16T19:56:54.768 INFO:teuthology.orchestra.run.smithi032.stdout:# file: . 2024-03-16T19:56:54.768 INFO:teuthology.orchestra.run.smithi032.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-16T19:56:54.768 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:54.769 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:56:54.769 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T19:56:54.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:54.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 Processor -- start 2024-03-16T19:56:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 -- start start 2024-03-16T19:56:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ed8151ce0 0x7f5ed80c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed80c3bd0 con 0x7f5ed80c3010 2024-03-16T19:56:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed81520c0 con 0x7f5ed8070210 2024-03-16T19:56:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.976+0000 7f5ee0547640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed80c3e50 con 0x7f5ed8151ce0 2024-03-16T19:56:54.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.980+0000 7f5eddae2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5ed8070210 0x7f5ed80c8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:46398/0 (socket says 172.21.15.32:46398) 2024-03-16T19:56:54.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:54.980+0000 7f5eddae2640 1 -- 172.21.15.32:0/1890733074 learned_addr learned my addr 172.21.15.32:0/1890733074 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:54.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.024+0000 7f5ede2e3640 1 --2- 172.21.15.32:0/1890733074 >> v2:172.21.15.32:3300/0 conn(0x7f5ed8151ce0 0x7f5ed80c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.032+0000 7f5ede2e3640 1 -- 172.21.15.32:0/1890733074 >> v1:172.21.15.59:6789/0 conn(0x7f5ed8070210 legacy=0x7f5ed80c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.032+0000 7f5ede2e3640 1 -- 172.21.15.32:0/1890733074 >> v1:172.21.15.32:6789/0 conn(0x7f5ed80c3010 legacy=0x7f5ed80c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.032+0000 7f5ede2e3640 1 -- 172.21.15.32:0/1890733074 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed806f170 con 0x7f5ed8151ce0 2024-03-16T19:56:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.036+0000 7f5ede2e3640 1 --2- 172.21.15.32:0/1890733074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed80c9100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40097c0 tx=0x7f5ed401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186ae358413c5bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5edd2e1640 1 -- 172.21.15.32:0/1890733074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed40a8990 con 0x7f5ed8151ce0 2024-03-16T19:56:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5edd2e1640 1 -- 172.21.15.32:0/1890733074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed40a8af0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5edd2e1640 1 -- 172.21.15.32:0/1890733074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed40a8e80 con 0x7f5ed8151ce0 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 -- 172.21.15.32:0/1890733074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 msgr2=0x7f5ed80c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 --2- 172.21.15.32:0/1890733074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed80c9100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40097c0 tx=0x7f5ed401ec90 comp rx=0 tx=0).stop 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 -- 172.21.15.32:0/1890733074 shutdown_connections 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 --2- 172.21.15.32:0/1890733074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed80c9100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 -- 172.21.15.32:0/1890733074 >> 172.21.15.32:0/1890733074 conn(0x7f5ed8070f30 msgr2=0x7f5ed8071340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 -- 172.21.15.32:0/1890733074 shutdown_connections 2024-03-16T19:56:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.048+0000 7f5ee0547640 1 -- 172.21.15.32:0/1890733074 wait complete. 2024-03-16T19:56:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 Processor -- start 2024-03-16T19:56:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 -- start start 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5ed8070210 0x7f5ed81569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed80c3010 0x7f5ed806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed81565d0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8150240 con 0x7f5ed80c3010 2024-03-16T19:56:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5ee0547640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed81f03c0 con 0x7f5ed8070210 2024-03-16T19:56:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5edeae4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5edeae4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39788/0 (socket says 172.21.15.32:39788) 2024-03-16T19:56:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.052+0000 7f5edeae4640 1 -- 172.21.15.32:0/2213934674 learned_addr learned my addr 172.21.15.32:0/2213934674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.056+0000 7f5edeae4640 1 -- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5ed8070210 msgr2=0x7f5ed81569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.056+0000 7f5edeae4640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5ed8070210 0x7f5ed81569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.056+0000 7f5edeae4640 1 -- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed80c3010 msgr2=0x7f5ed806d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.056+0000 7f5edeae4640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed80c3010 0x7f5ed806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.056+0000 7f5edeae4640 1 -- 172.21.15.32:0/2213934674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4009420 con 0x7f5ed8151ce0 2024-03-16T19:56:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.060+0000 7f5edeae4640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000c770 tx=0x7f5ed000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0003b80 con 0x7f5ed8151ce0 2024-03-16T19:56:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed0003ce0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed81f08d0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed000aa90 con 0x7f5ed8151ce0 2024-03-16T19:56:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed81f05f0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.064+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed8156710 con 0x7f5ed8151ce0 2024-03-16T19:56:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.088+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5ed000abf0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.088+0000 7f5eceffd640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 0x7f5ebc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.088+0000 7f5ede2e3640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 0x7f5ebc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.088+0000 7f5ede2e3640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 0x7f5ebc069380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40b2000 tx=0x7f5ed4004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:55.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.088+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5ed0056420 con 0x7f5ed8151ce0 2024-03-16T19:56:55.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.392+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5ed0051fc0 con 0x7f5ed8151ce0 2024-03-16T19:56:55.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.580+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed8156710 con 0x7f5ed8151ce0 2024-03-16T19:56:55.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.636+0000 7f5eceffd640 1 -- 172.21.15.32:0/2213934674 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f5ed8156710 con 0x7f5ed8151ce0 2024-03-16T19:56:55.610 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:56:55.610 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 msgr2=0x7f5ebc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 0x7f5ebc069380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40b2000 tx=0x7f5ed4004740 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 msgr2=0x7f5ed814a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000c770 tx=0x7f5ed000cc30 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 shutdown_connections 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ebc066ec0 0x7f5ebc069380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed8151ce0 0x7f5ed814a910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed80c3010 0x7f5ed806d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 --2- 172.21.15.32:0/2213934674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5ed8070210 0x7f5ed81569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 >> 172.21.15.32:0/2213934674 conn(0x7f5ed8070f30 msgr2=0x7f5ed81e6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 shutdown_connections 2024-03-16T19:56:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.640+0000 7f5ee0547640 1 -- 172.21.15.32:0/2213934674 wait complete. 2024-03-16T19:56:55.631 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:56:31.637719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:56:33.839961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T19:56:40.663264+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T19:56:55.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:56:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.780+0000 7fe123b4f640 1 Processor -- start 2024-03-16T19:56:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.780+0000 7fe123b4f640 1 -- start start 2024-03-16T19:56:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe123b4f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe11c0c7040 0x7fe11c0c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe123b4f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c067320 con 0x7fe11c151cc0 2024-03-16T19:56:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe123b4f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0c9f30 con 0x7fe11c158070 2024-03-16T19:56:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe123b4f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c06bbc0 con 0x7fe11c0c7040 2024-03-16T19:56:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe1218eb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe11c151cc0 0x7fe11c0c9820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58464/0 (socket says 172.21.15.32:58464) 2024-03-16T19:56:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe1218eb640 1 -- 172.21.15.32:0/1406252601 learned_addr learned my addr 172.21.15.32:0/1406252601 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.784+0000 7fe1210ea640 1 --2- 172.21.15.32:0/1406252601 >> v2:172.21.15.32:3300/0 conn(0x7fe11c0c7040 0x7fe11c0c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.788+0000 7fe1210ea640 1 -- 172.21.15.32:0/1406252601 >> v1:172.21.15.59:6789/0 conn(0x7fe11c158070 legacy=0x7fe11c1584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.788+0000 7fe1210ea640 1 -- 172.21.15.32:0/1406252601 >> v1:172.21.15.32:6789/0 conn(0x7fe11c151cc0 legacy=0x7fe11c0c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.788+0000 7fe1210ea640 1 -- 172.21.15.32:0/1406252601 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c0704d0 con 0x7fe11c0c7040 2024-03-16T19:56:55.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.792+0000 7fe1210ea640 1 --2- 172.21.15.32:0/1406252601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c0c74b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe1180097c0 tx=0x7fe11801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6878e2d610709e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.796+0000 7fe1208e9640 1 -- 172.21.15.32:0/1406252601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1180a8050 con 0x7fe11c0c7040 2024-03-16T19:56:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.796+0000 7fe1208e9640 1 -- 172.21.15.32:0/1406252601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1180ac5f0 con 0x7fe11c0c7040 2024-03-16T19:56:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.796+0000 7fe1208e9640 1 -- 172.21.15.32:0/1406252601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1180ac980 con 0x7fe11c0c7040 2024-03-16T19:56:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- 172.21.15.32:0/1406252601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 msgr2=0x7fe11c0c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 --2- 172.21.15.32:0/1406252601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c0c74b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe1180097c0 tx=0x7fe11801c4a0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- 172.21.15.32:0/1406252601 shutdown_connections 2024-03-16T19:56:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 --2- 172.21.15.32:0/1406252601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c0c74b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- 172.21.15.32:0/1406252601 >> 172.21.15.32:0/1406252601 conn(0x7fe11c066ea0 msgr2=0x7fe11c0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- 172.21.15.32:0/1406252601 shutdown_connections 2024-03-16T19:56:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- 172.21.15.32:0/1406252601 wait complete. 2024-03-16T19:56:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 Processor -- start 2024-03-16T19:56:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- start start 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe11c151cc0 0x7fe11c154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe11c158070 0x7fe11c154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c1540e0 con 0x7fe11c0c7040 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c152fe0 con 0x7fe11c151cc0 2024-03-16T19:56:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.800+0000 7fe123b4f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0cbe60 con 0x7fe11c158070 2024-03-16T19:56:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.804+0000 7fe1218eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.804+0000 7fe1218eb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39798/0 (socket says 172.21.15.32:39798) 2024-03-16T19:56:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.804+0000 7fe1218eb640 1 -- 172.21.15.32:0/188407324 learned_addr learned my addr 172.21.15.32:0/188407324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.804+0000 7fe1210ea640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe11c151cc0 0x7fe11c154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.804+0000 7fe1220ec640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe11c158070 0x7fe11c154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 -- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe11c158070 msgr2=0x7fe11c154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe11c158070 0x7fe11c154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 -- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe11c151cc0 msgr2=0x7fe11c154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe11c151cc0 0x7fe11c154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 -- 172.21.15.32:0/188407324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118009420 con 0x7fe11c0c7040 2024-03-16T19:56:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.808+0000 7fe1218eb640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c500 tx=0x7fe10c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.812+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c005cf0 con 0x7fe11c0c7040 2024-03-16T19:56:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.812+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c0cc090 con 0x7fe11c0c7040 2024-03-16T19:56:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.812+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c1526b0 con 0x7fe11c0c7040 2024-03-16T19:56:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:55.812+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c068000 con 0x7fe11c0c7040 2024-03-16T19:56:55.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.028+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c0011f0 con 0x7fe11c0c7040 2024-03-16T19:56:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.072+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0072f0 con 0x7fe11c0c7040 2024-03-16T19:56:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.084+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe10c007510 con 0x7fe11c0c7040 2024-03-16T19:56:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.084+0000 7fe1127fc640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 0x7fe100069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.084+0000 7fe1210ea640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 0x7fe100069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.088+0000 7fe1210ea640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 0x7fe100069300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe1180ae000 tx=0x7fe118008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:56.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.100+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe10c055420 con 0x7fe11c0c7040 2024-03-16T19:56:56.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.100+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe10c0af210 con 0x7fe11c0c7040 2024-03-16T19:56:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.272+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe11c06cd10 con 0x7fe11c0c7040 2024-03-16T19:56:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe1127fc640 1 -- 172.21.15.32:0/188407324 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe10c0170a0 con 0x7fe11c0c7040 2024-03-16T19:56:56.252 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:56:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 msgr2=0x7fe100069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 0x7fe100069300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe1180ae000 tx=0x7fe118008040 comp rx=0 tx=0).stop 2024-03-16T19:56:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 msgr2=0x7fe11c153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c500 tx=0x7fe10c00c9c0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 shutdown_connections 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe100066e40 0x7fe100069300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe11c158070 0x7fe11c154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe11c151cc0 0x7fe11c154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 --2- 172.21.15.32:0/188407324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe11c0c7040 0x7fe11c153ba0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.280+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 >> 172.21.15.32:0/188407324 conn(0x7fe11c066ea0 msgr2=0x7fe11c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.284+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 shutdown_connections 2024-03-16T19:56:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.284+0000 7fe123b4f640 1 -- 172.21.15.32:0/188407324 wait complete. 2024-03-16T19:56:56.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:56:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 Processor -- start 2024-03-16T19:56:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 -- start start 2024-03-16T19:56:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa220071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa220073280 con 0x7fa22015d8f0 2024-03-16T19:56:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa220073520 con 0x7fa22015d510 2024-03-16T19:56:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa225a3f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200c12d0 con 0x7fa2200c7ea0 2024-03-16T19:56:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa21f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa22015d8f0 0x7fa22015dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58470/0 (socket says 172.21.15.32:58470) 2024-03-16T19:56:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.416+0000 7fa21f7fe640 1 -- 172.21.15.32:0/644253960 learned_addr learned my addr 172.21.15.32:0/644253960 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:56.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.420+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801699714 0 0) 0x7fa220073280 con 0x7fa22015d8f0 2024-03-16T19:56:56.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.420+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208003530 con 0x7fa22015d8f0 2024-03-16T19:56:56.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.420+0000 7fa21e7fc640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa220071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373731596 0 0) 0x7fa208003530 con 0x7fa22015d8f0 2024-03-16T19:56:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 msgr2=0x7fa220071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa220071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 >> v1:172.21.15.59:6789/0 conn(0x7fa22015d510 legacy=0x7fa2200c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200c7120 con 0x7fa22015d8f0 2024-03-16T19:56:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fa218001a00 con 0x7fa22015d8f0 2024-03-16T19:56:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 >> v1:172.21.15.32:6789/0 conn(0x7fa22015d8f0 legacy=0x7fa22015dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2080039b0 0x7fa208003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa2080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa208004d10 0x7fa208005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200c12d0 con 0x7fa208004d10 2024-03-16T19:56:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20800bb20 con 0x7fa2080039b0 2024-03-16T19:56:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.424+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20800bcb0 con 0x7fa2200c7ea0 2024-03-16T19:56:56.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.428+0000 7fa21e7fc640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa2080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.432+0000 7fa21effd640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2080039b0 0x7fa208003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.436+0000 7fa21e7fc640 1 -- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2080039b0 msgr2=0x7fa208003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.436+0000 7fa21e7fc640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2080039b0 0x7fa208003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.436+0000 7fa21e7fc640 1 -- 172.21.15.32:0/644253960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa208004d10 msgr2=0x7fa208005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.436+0000 7fa21e7fc640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa208004d10 0x7fa208005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.436+0000 7fa21e7fc640 1 -- 172.21.15.32:0/644253960 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208003530 con 0x7fa2200c7ea0 2024-03-16T19:56:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.444+0000 7fa21e7fc640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa2080047d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa21400ba30 tx=0x7fa21401c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=406c246bf5e6d240 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140a9680 con 0x7fa2200c7ea0 2024-03-16T19:56:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2140a97e0 con 0x7fa2200c7ea0 2024-03-16T19:56:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa21dffb640 1 -- 172.21.15.32:0/644253960 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140a9b90 con 0x7fa2200c7ea0 2024-03-16T19:56:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 -- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 msgr2=0x7fa2080047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 --2- 172.21.15.32:0/644253960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c7ea0 0x7fa2080047d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa21400ba30 tx=0x7fa21401c6a0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa21f7fe640 1 -- 172.21.15.32:0/644253960 reap_dead start 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 -- 172.21.15.32:0/644253960 shutdown_connections 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 -- 172.21.15.32:0/644253960 >> 172.21.15.32:0/644253960 conn(0x7fa22006f3a0 msgr2=0x7fa22006f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 -- 172.21.15.32:0/644253960 shutdown_connections 2024-03-16T19:56:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.452+0000 7fa225a3f640 1 -- 172.21.15.32:0/644253960 wait complete. 2024-03-16T19:56:56.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 Processor -- start 2024-03-16T19:56:56.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 -- start start 2024-03-16T19:56:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2200c7ea0 0x7fa22014fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa22015aae0 0x7fa2200c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2201efdd0 con 0x7fa2200c7ea0 2024-03-16T19:56:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200c5440 con 0x7fa22015aae0 2024-03-16T19:56:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.456+0000 7fa225a3f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa220150140 con 0x7fa2200c4f60 2024-03-16T19:56:56.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.460+0000 7fa21f7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.460+0000 7fa21effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2200c7ea0 0x7fa22014fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.460+0000 7fa21e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa22015aae0 0x7fa2200c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.460+0000 7fa21f7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:38898/0 (socket says 172.21.15.32:38898) 2024-03-16T19:56:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.460+0000 7fa21f7fe640 1 -- 172.21.15.32:0/2996847147 learned_addr learned my addr 172.21.15.32:0/2996847147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.476+0000 7fa21f7fe640 1 -- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa22015aae0 msgr2=0x7fa2200c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.476+0000 7fa21f7fe640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa22015aae0 0x7fa2200c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.476+0000 7fa21f7fe640 1 -- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2200c7ea0 msgr2=0x7fa22014fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.476+0000 7fa21f7fe640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2200c7ea0 0x7fa22014fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.476+0000 7fa21f7fe640 1 -- 172.21.15.32:0/2996847147 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2201effe0 con 0x7fa2200c4f60 2024-03-16T19:56:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.492+0000 7fa21f7fe640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa22015dcd0 tx=0x7fa21800cee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.600+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800a610 con 0x7fa2200c4f60 2024-03-16T19:56:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.600+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21800a770 con 0x7fa2200c4f60 2024-03-16T19:56:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.604+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2201f02c0 con 0x7fa2200c4f60 2024-03-16T19:56:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.604+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800ab20 con 0x7fa2200c4f60 2024-03-16T19:56:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.604+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2201f0880 con 0x7fa2200c4f60 2024-03-16T19:56:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.604+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22006c140 con 0x7fa2200c4f60 2024-03-16T19:56:56.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.988+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa21800ac80 con 0x7fa2200c4f60 2024-03-16T19:56:56.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.988+0000 7fa207fff640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 0x7fa20c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:56.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.988+0000 7fa21effd640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 0x7fa20c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.988+0000 7fa21effd640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 0x7fa20c069350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa210007fd0 tx=0x7fa210009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:56.996+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa218055920 con 0x7fa2200c4f60 2024-03-16T19:56:56.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.024+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa2180514c0 con 0x7fa2200c4f60 2024-03-16T19:56:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.180+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa220068000 con 0x7fa2200c4f60 2024-03-16T19:56:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.244+0000 7fa207fff640 1 -- 172.21.15.32:0/2996847147 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa218054b80 con 0x7fa2200c4f60 2024-03-16T19:56:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 msgr2=0x7fa20c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 0x7fa20c069350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa210007fd0 tx=0x7fa210009040 comp rx=0 tx=0).stop 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 msgr2=0x7fa22015c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa22015dcd0 tx=0x7fa21800cee0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 shutdown_connections 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa20c066e90 0x7fa20c069350 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa2200c4f60 0x7fa22015c340 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa22015aae0 0x7fa2200c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 --2- 172.21.15.32:0/2996847147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2200c7ea0 0x7fa22014fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 >> 172.21.15.32:0/2996847147 conn(0x7fa22006f3a0 msgr2=0x7fa2200c6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 shutdown_connections 2024-03-16T19:56:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.248+0000 7fa225a3f640 1 -- 172.21.15.32:0/2996847147 wait complete. 2024-03-16T19:56:57.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:56:57.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 Processor -- start 2024-03-16T19:56:57.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 -- start start 2024-03-16T19:56:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71800c7040 0x7f71800c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7180158070 0x7f71801584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7180067320 con 0x7f7180151cc0 2024-03-16T19:56:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718006bbc0 con 0x7f71800c7040 2024-03-16T19:56:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f7184ef3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7180157c20 con 0x7f7180158070 2024-03-16T19:56:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f717dd9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71800c7040 0x7f71800c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f717dd9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71800c7040 0x7f71800c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39830/0 (socket says 172.21.15.32:39830) 2024-03-16T19:56:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f717dd9b640 1 -- 172.21.15.32:0/360520173 learned_addr learned my addr 172.21.15.32:0/360520173 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.392+0000 7f717ed9d640 1 --2- 172.21.15.32:0/360520173 >> v2:172.21.15.59:3300/0 conn(0x7f7180158070 0x7f71801584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.396+0000 7f717dd9b640 1 -- 172.21.15.32:0/360520173 >> v2:172.21.15.59:3300/0 conn(0x7f7180158070 msgr2=0x7f71801584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.396+0000 7f717dd9b640 1 --2- 172.21.15.32:0/360520173 >> v2:172.21.15.59:3300/0 conn(0x7f7180158070 0x7f71801584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.396+0000 7f717dd9b640 1 -- 172.21.15.32:0/360520173 >> v1:172.21.15.59:6789/0 conn(0x7f7180151cc0 legacy=0x7f71800c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.396+0000 7f717dd9b640 1 -- 172.21.15.32:0/360520173 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800c9f30 con 0x7f71800c7040 2024-03-16T19:56:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.396+0000 7f717dd9b640 1 --2- 172.21.15.32:0/360520173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71800c74b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f71800c8e40 tx=0x7f716801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee91439f4ba4a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.400+0000 7f717d59a640 1 -- 172.21.15.32:0/360520173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680aa640 con 0x7f71800c7040 2024-03-16T19:56:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.400+0000 7f717d59a640 1 -- 172.21.15.32:0/360520173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71680aa7a0 con 0x7f71800c7040 2024-03-16T19:56:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.400+0000 7f717d59a640 1 -- 172.21.15.32:0/360520173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680aab30 con 0x7f71800c7040 2024-03-16T19:56:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- 172.21.15.32:0/360520173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 msgr2=0x7f71800c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- 172.21.15.32:0/360520173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71800c74b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f71800c8e40 tx=0x7f716801c4e0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- 172.21.15.32:0/360520173 shutdown_connections 2024-03-16T19:56:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- 172.21.15.32:0/360520173 >> v2:172.21.15.59:3300/0 conn(0x7f7180158070 0x7f71801584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- 172.21.15.32:0/360520173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71800c74b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- 172.21.15.32:0/360520173 >> 172.21.15.32:0/360520173 conn(0x7f7180066ea0 msgr2=0x7f71800721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- 172.21.15.32:0/360520173 shutdown_connections 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- 172.21.15.32:0/360520173 wait complete. 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 Processor -- start 2024-03-16T19:56:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- start start 2024-03-16T19:56:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7180151cc0 0x7f71800cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7180158070 0x7f71800cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7180142190 con 0x7f71800c7040 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71800ca670 con 0x7f7180158070 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f7184ef3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7180157440 con 0x7f7180151cc0 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f717e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f717e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39842/0 (socket says 172.21.15.32:39842) 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f717e59c640 1 -- 172.21.15.32:0/4222518661 learned_addr learned my addr 172.21.15.32:0/4222518661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f717dd9b640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7180151cc0 0x7f71800cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.404+0000 7f717ed9d640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7180158070 0x7f71800cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.408+0000 7f717e59c640 1 -- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7180151cc0 msgr2=0x7f71800cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.408+0000 7f717e59c640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7180151cc0 0x7f71800cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.408+0000 7f717e59c640 1 -- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7180158070 msgr2=0x7f71800cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.408+0000 7f717e59c640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7180158070 0x7f71800cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.408+0000 7f717e59c640 1 -- 172.21.15.32:0/4222518661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71680094d0 con 0x7f71800c7040 2024-03-16T19:56:57.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.412+0000 7f717e59c640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f717800c7c0 tx=0x7f717800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:57.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717800aa50 con 0x7f71800c7040 2024-03-16T19:56:57.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717800abb0 con 0x7f71800c7040 2024-03-16T19:56:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178012420 con 0x7f71800c7040 2024-03-16T19:56:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71801f6040 con 0x7f71800c7040 2024-03-16T19:56:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71801f6600 con 0x7f71800c7040 2024-03-16T19:56:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.416+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7180068000 con 0x7f71800c7040 2024-03-16T19:56:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.692+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7178006b60 con 0x7f71800c7040 2024-03-16T19:56:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.696+0000 7f7176ffd640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 0x7f7160069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.696+0000 7f717dd9b640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 0x7f7160069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.696+0000 7f717dd9b640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 0x7f7160069350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f71680ae000 tx=0x7f71680b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.712+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7178055cc0 con 0x7f71800c7040 2024-03-16T19:56:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.716+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7178051f60 con 0x7f71800c7040 2024-03-16T19:56:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.876+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71800ca440 con 0x7f71800c7040 2024-03-16T19:56:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7176ffd640 1 -- 172.21.15.32:0/4222518661 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7178016070 con 0x7f71800c7040 2024-03-16T19:56:57.857 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 msgr2=0x7f7160069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 0x7f7160069350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f71680ae000 tx=0x7f71680b5040 comp rx=0 tx=0).stop 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 msgr2=0x7f71801435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f717800c7c0 tx=0x7f717800cc80 comp rx=0 tx=0).stop 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 shutdown_connections 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7160066e90 0x7f7160069350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7180158070 0x7f71800cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7180151cc0 0x7f71800cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 --2- 172.21.15.32:0/4222518661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71800c7040 0x7f71801435f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 >> 172.21.15.32:0/4222518661 conn(0x7f7180066ea0 msgr2=0x7f718006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 shutdown_connections 2024-03-16T19:56:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:57.884+0000 7f7184ef3640 1 -- 172.21.15.32:0/4222518661 wait complete. 2024-03-16T19:56:57.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T19:56:57.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.016+0000 7f8d776b5640 1 Processor -- start 2024-03-16T19:56:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.016+0000 7f8d776b5640 1 -- start start 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.016+0000 7f8d776b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8d701540e0 0x7f8d70072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d776b5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 0x7f8d7015d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d776b5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7015de90 con 0x7f8d70070430 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d776b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7015abe0 con 0x7f8d701540e0 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d776b5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70070b90 con 0x7f8d7015d4c0 2024-03-16T19:56:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d74c50640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d70070430 0x7f8d7006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58476/0 (socket says 172.21.15.32:58476) 2024-03-16T19:56:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d74c50640 1 -- 172.21.15.32:0/2164588752 learned_addr learned my addr 172.21.15.32:0/2164588752 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103908165 0 0) 0x7f8d7015de90 con 0x7f8d70070430 2024-03-16T19:56:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.020+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c003530 con 0x7f8d70070430 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291896554 0 0) 0x7f8d5c003530 con 0x7f8d70070430 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 msgr2=0x7f8d7015d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 0x7f8d7015d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 >> v2:172.21.15.59:3300/0 conn(0x7f8d701540e0 msgr2=0x7f8d70072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 --2- 172.21.15.32:0/2164588752 >> v2:172.21.15.59:3300/0 conn(0x7f8d701540e0 0x7f8d70072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70159fd0 con 0x7f8d70070430 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f8d60001680 con 0x7f8d70070430 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 >> v1:172.21.15.32:6789/0 conn(0x7f8d70070430 legacy=0x7f8d7006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 0x7f8d5c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c0040e0 0x7f8d5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d5c004e30 0x7f8d5c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7015abe0 con 0x7f8d5c0040e0 2024-03-16T19:56:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70070b90 con 0x7f8d5c004e30 2024-03-16T19:56:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.024+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c005640 con 0x7f8d7015d4c0 2024-03-16T19:56:58.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.028+0000 7f8d74c50640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c0040e0 0x7f8d5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.032+0000 7f8d74c50640 1 -- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 msgr2=0x7f8d5c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.032+0000 7f8d74c50640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d7015d4c0 0x7f8d5c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.032+0000 7f8d74c50640 1 -- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d5c004e30 msgr2=0x7f8d5c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.032+0000 7f8d74c50640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d5c004e30 0x7f8d5c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.032+0000 7f8d74c50640 1 -- 172.21.15.32:0/2164588752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c003530 con 0x7f8d5c0040e0 2024-03-16T19:56:58.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.036+0000 7f8d74c50640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c0040e0 0x7f8d5c0047d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c160 tx=0x7f8d6001cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3345b5b0fe20d00e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600058e0 con 0x7f8d5c0040e0 2024-03-16T19:56:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60005a40 con 0x7f8d5c0040e0 2024-03-16T19:56:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d67fff640 1 -- 172.21.15.32:0/2164588752 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60005e30 con 0x7f8d5c0040e0 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- 172.21.15.32:0/2164588752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c0040e0 msgr2=0x7f8d5c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 --2- 172.21.15.32:0/2164588752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c0040e0 0x7f8d5c0047d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c160 tx=0x7f8d6001cb40 comp rx=0 tx=0).stop 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d75c52640 1 -- 172.21.15.32:0/2164588752 reap_dead start 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- 172.21.15.32:0/2164588752 shutdown_connections 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- 172.21.15.32:0/2164588752 >> 172.21.15.32:0/2164588752 conn(0x7f8d700cc7a0 msgr2=0x7f8d700ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- 172.21.15.32:0/2164588752 shutdown_connections 2024-03-16T19:56:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- 172.21.15.32:0/2164588752 wait complete. 2024-03-16T19:56:58.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 Processor -- start 2024-03-16T19:56:58.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.040+0000 7f8d776b5640 1 -- start start 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d5c0040e0 0x7f8d701475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d701f32c0 0x7f8d70158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70156930 con 0x7f8d70154e90 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c4c40 con 0x7f8d701f32c0 2024-03-16T19:56:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d776b5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70155360 con 0x7f8d5c0040e0 2024-03-16T19:56:58.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d74c50640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d74c50640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39874/0 (socket says 172.21.15.32:39874) 2024-03-16T19:56:58.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.044+0000 7f8d74c50640 1 -- 172.21.15.32:0/87241625 learned_addr learned my addr 172.21.15.32:0/87241625 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.048+0000 7f8d74c50640 1 -- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d5c0040e0 msgr2=0x7f8d701475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.048+0000 7f8d74c50640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d5c0040e0 0x7f8d701475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.048+0000 7f8d74c50640 1 -- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d701f32c0 msgr2=0x7f8d70158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.048+0000 7f8d74c50640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d701f32c0 0x7f8d70158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.048+0000 7f8d74c50640 1 -- 172.21.15.32:0/87241625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70155620 con 0x7f8d70154e90 2024-03-16T19:56:58.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.052+0000 7f8d74c50640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d60000c10 tx=0x7f8d5c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600a5b70 con 0x7f8d70154e90 2024-03-16T19:56:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d600a5cd0 con 0x7f8d70154e90 2024-03-16T19:56:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60004a10 con 0x7f8d70154e90 2024-03-16T19:56:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70155900 con 0x7f8d70154e90 2024-03-16T19:56:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7015b150 con 0x7f8d70154e90 2024-03-16T19:56:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.056+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d700c4000 con 0x7f8d70154e90 2024-03-16T19:56:58.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.352+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8d60004b70 con 0x7f8d70154e90 2024-03-16T19:56:58.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.352+0000 7f8d65ffb640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 0x7f8d54069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.352+0000 7f8d75451640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 0x7f8d54069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.352+0000 7f8d75451640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 0x7f8d54069380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c006fd0 tx=0x7f8d6c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.388+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8d600a5e40 con 0x7f8d70154e90 2024-03-16T19:56:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.388+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8d600ebcb0 con 0x7f8d70154e90 2024-03-16T19:56:58.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.560+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8d70068000 con 0x7f8d70154e90 2024-03-16T19:56:58.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.568+0000 7f8d65ffb640 1 -- 172.21.15.32:0/87241625 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8d600ef370 con 0x7f8d70154e90 2024-03-16T19:56:58.542 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:56:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 msgr2=0x7f8d54069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 0x7f8d54069380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c006fd0 tx=0x7f8d6c006d80 comp rx=0 tx=0).stop 2024-03-16T19:56:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 msgr2=0x7f8d70157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d60000c10 tx=0x7f8d5c005800 comp rx=0 tx=0).stop 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 shutdown_connections 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8d54066ec0 0x7f8d54069380 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d701f32c0 0x7f8d70158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70154e90 0x7f8d70157d30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 --2- 172.21.15.32:0/87241625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8d5c0040e0 0x7f8d701475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 >> 172.21.15.32:0/87241625 conn(0x7f8d700cc7a0 msgr2=0x7f8d7006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 shutdown_connections 2024-03-16T19:56:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.572+0000 7f8d776b5640 1 -- 172.21.15.32:0/87241625 wait complete. 2024-03-16T19:56:58.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T19:56:58.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 Processor -- start 2024-03-16T19:56:58.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 -- start start 2024-03-16T19:56:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a741541a0 0x7f1a7406b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a7415a990 0x7f1a740cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a74154580 con 0x7f1a7406ba90 2024-03-16T19:56:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a74159e30 con 0x7f1a741541a0 2024-03-16T19:56:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a7b6c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7415d5c0 con 0x7f1a7415a990 2024-03-16T19:56:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a79461640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a741541a0 0x7f1a7406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a79461640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a741541a0 0x7f1a7406b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39888/0 (socket says 172.21.15.32:39888) 2024-03-16T19:56:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a79461640 1 -- 172.21.15.32:0/800154560 learned_addr learned my addr 172.21.15.32:0/800154560 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.700+0000 7f1a79c62640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a7415a990 0x7f1a740cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.704+0000 7f1a79461640 1 -- 172.21.15.32:0/800154560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a7415a990 msgr2=0x7f1a740cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.704+0000 7f1a79461640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a7415a990 0x7f1a740cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.704+0000 7f1a79461640 1 -- 172.21.15.32:0/800154560 >> v1:172.21.15.59:6789/0 conn(0x7f1a7406ba90 legacy=0x7f1a7415a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.704+0000 7f1a79461640 1 -- 172.21.15.32:0/800154560 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7415d8a0 con 0x7f1a741541a0 2024-03-16T19:56:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.708+0000 7f1a79461640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a741541a0 0x7f1a7406b550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009870 tx=0x7f1a7001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae9aae8fa035edb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a6bfff640 1 -- 172.21.15.32:0/800154560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a700a8a50 con 0x7f1a741541a0 2024-03-16T19:56:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a6bfff640 1 -- 172.21.15.32:0/800154560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a700a8bb0 con 0x7f1a741541a0 2024-03-16T19:56:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a6bfff640 1 -- 172.21.15.32:0/800154560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a700b0440 con 0x7f1a741541a0 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/800154560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a741541a0 msgr2=0x7f1a7406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a741541a0 0x7f1a7406b550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009870 tx=0x7f1a7001ed50 comp rx=0 tx=0).stop 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/800154560 shutdown_connections 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a7415a990 0x7f1a740cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/800154560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a741541a0 0x7f1a7406b550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/800154560 >> 172.21.15.32:0/800154560 conn(0x7f1a74066ea0 msgr2=0x7f1a740721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/800154560 shutdown_connections 2024-03-16T19:56:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/800154560 wait complete. 2024-03-16T19:56:58.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 Processor -- start 2024-03-16T19:56:58.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.712+0000 7f1a7b6c5640 1 -- start start 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 0x7f1a7414ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 0x7f1a740c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a7415a990 0x7f1a741f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414e230 con 0x7f1a7406ba90 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414d170 con 0x7f1a7415a990 2024-03-16T19:56:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a7b6c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414ea40 con 0x7f1a741541a0 2024-03-16T19:56:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a78c60640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 0x7f1a740c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a79c62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a7415a990 0x7f1a741f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a78c60640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 0x7f1a740c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:38930/0 (socket says 172.21.15.32:38930) 2024-03-16T19:56:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.716+0000 7f1a78c60640 1 -- 172.21.15.32:0/1295822561 learned_addr learned my addr 172.21.15.32:0/1295822561 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:58.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.720+0000 7f1a79461640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 0x7f1a7414ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.724+0000 7f1a79461640 1 -- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 msgr2=0x7f1a740c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.724+0000 7f1a79461640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 0x7f1a740c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.724+0000 7f1a79461640 1 -- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a7415a990 msgr2=0x7f1a741f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:58.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.724+0000 7f1a79461640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a7415a990 0x7f1a741f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:58.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.724+0000 7f1a79461640 1 -- 172.21.15.32:0/1295822561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a700094d0 con 0x7f1a7406ba90 2024-03-16T19:56:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.728+0000 7f1a79461640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 0x7f1a7414ae00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a700af000 tx=0x7f1a7001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a700be070 con 0x7f1a7406ba90 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a700ba400 con 0x7f1a7406ba90 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a700ba7d0 con 0x7f1a7406ba90 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7414d430 con 0x7f1a7406ba90 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7414e760 con 0x7f1a7406ba90 2024-03-16T19:56:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.732+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a741f29e0 con 0x7f1a7406ba90 2024-03-16T19:56:58.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.752+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1a700c3410 con 0x7f1a7406ba90 2024-03-16T19:56:58.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.752+0000 7f1a69ffb640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 0x7f1a58069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:58.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.752+0000 7f1a78c60640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 0x7f1a58069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:58.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:58.752+0000 7f1a78c60640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 0x7f1a58069350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a74070440 tx=0x7f1a64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.076+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1a700f3980 con 0x7f1a7406ba90 2024-03-16T19:56:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.076+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1a700efc20 con 0x7f1a7406ba90 2024-03-16T19:56:59.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.264+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1a74068000 con 0x7f1a7406ba90 2024-03-16T19:56:59.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a69ffb640 1 -- 172.21.15.32:0/1295822561 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1a700b5020 con 0x7f1a7406ba90 2024-03-16T19:56:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 msgr2=0x7f1a58069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 0x7f1a58069350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a74070440 tx=0x7f1a64009040 comp rx=0 tx=0).stop 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 msgr2=0x7f1a7414ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 0x7f1a7414ae00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a700af000 tx=0x7f1a7001baf0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 shutdown_connections 2024-03-16T19:56:59.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a58066e90 0x7f1a58069350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a7415a990 0x7f1a741f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a741541a0 0x7f1a740c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 --2- 172.21.15.32:0/1295822561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a7406ba90 0x7f1a7414ae00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.324+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 >> 172.21.15.32:0/1295822561 conn(0x7f1a74066ea0 msgr2=0x7f1a740c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.328+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 shutdown_connections 2024-03-16T19:56:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.328+0000 7f1a7b6c5640 1 -- 172.21.15.32:0/1295822561 wait complete. 2024-03-16T19:56:59.314 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-16T19:56:59.315 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T19:56:59.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:56:59.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.516+0000 7f62dfc48640 1 Processor -- start 2024-03-16T19:56:59.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.516+0000 7f62dfc48640 1 -- start start 2024-03-16T19:56:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dfc48640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62d8150c40 0x7f62d8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dfc48640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 0x7f62d8070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dfc48640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d8151020 con 0x7f62d80708a0 2024-03-16T19:56:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dfc48640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d815a000 con 0x7f62d8150c40 2024-03-16T19:56:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dfc48640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d8072da0 con 0x7f62d80725e0 2024-03-16T19:56:59.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62de1e5640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f62d80708a0 0x7f62d815a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49538/0 (socket says 172.21.15.32:49538) 2024-03-16T19:56:59.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62de1e5640 1 -- 172.21.15.32:0/2999360331 learned_addr learned my addr 172.21.15.32:0/2999360331 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:56:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dd9e4640 1 --2- 172.21.15.32:0/2999360331 >> v2:172.21.15.59:3300/0 conn(0x7f62d8150c40 0x7f62d8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.520+0000 7f62dd1e3640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 0x7f62d8070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622467029 0 0) 0x7f62d8151020 con 0x7f62d80708a0 2024-03-16T19:56:59.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4003530 con 0x7f62d80708a0 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757250407 0 0) 0x7f62c4003530 con 0x7f62d80708a0 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 msgr2=0x7f62d8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 0x7f62d8070360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 >> v2:172.21.15.59:3300/0 conn(0x7f62d8150c40 msgr2=0x7f62d8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 --2- 172.21.15.32:0/2999360331 >> v2:172.21.15.59:3300/0 conn(0x7f62d8150c40 0x7f62d8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d815e310 con 0x7f62d80708a0 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f62d0001a20 con 0x7f62d80708a0 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 >> v1:172.21.15.32:6789/0 conn(0x7f62d80708a0 legacy=0x7f62d815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62c4003980 0x7f62c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 0x7f62c40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62c4004d10 0x7f62c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d815a000 con 0x7f62c4004d10 2024-03-16T19:56:59.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d8072da0 con 0x7f62c4003980 2024-03-16T19:56:59.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.524+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c400b990 con 0x7f62d80725e0 2024-03-16T19:56:59.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.528+0000 7f62de1e5640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62c4004d10 0x7f62c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.532+0000 7f62de1e5640 1 -- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 msgr2=0x7f62c40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.532+0000 7f62de1e5640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d80725e0 0x7f62c40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.532+0000 7f62de1e5640 1 -- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62c4003980 msgr2=0x7f62c4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:56:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.532+0000 7f62de1e5640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62c4003980 0x7f62c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.532+0000 7f62de1e5640 1 -- 172.21.15.32:0/2999360331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d815e310 con 0x7f62c4004d10 2024-03-16T19:56:59.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.536+0000 7f62de1e5640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62c4004d10 0x7f62c4005190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62d000a8c0 tx=0x7f62d001d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51838b83e06c3a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00ad630 con 0x7f62c4004d10 2024-03-16T19:56:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d00a8070 con 0x7f62c4004d10 2024-03-16T19:56:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dc9e2640 1 -- 172.21.15.32:0/2999360331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00ada50 con 0x7f62c4004d10 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- 172.21.15.32:0/2999360331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62c4004d10 msgr2=0x7f62c4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 --2- 172.21.15.32:0/2999360331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62c4004d10 0x7f62c4005190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62d000a8c0 tx=0x7f62d001d380 comp rx=0 tx=0).stop 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62de1e5640 1 -- 172.21.15.32:0/2999360331 reap_dead start 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- 172.21.15.32:0/2999360331 shutdown_connections 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- 172.21.15.32:0/2999360331 >> 172.21.15.32:0/2999360331 conn(0x7f62d806c0d0 msgr2=0x7f62d806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- 172.21.15.32:0/2999360331 shutdown_connections 2024-03-16T19:56:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- 172.21.15.32:0/2999360331 wait complete. 2024-03-16T19:56:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 Processor -- start 2024-03-16T19:56:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 -- start start 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.540+0000 7f62dfc48640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d8147840 0x7f62d814c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dfc48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d814c970 0x7f62d8148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dfc48640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d81efd80 con 0x7f62d8147460 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dfc48640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d81f0800 con 0x7f62d814c970 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dfc48640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d814bc80 con 0x7f62d8147840 2024-03-16T19:56:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dd9e4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dd9e4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54866/0 (socket says 172.21.15.32:54866) 2024-03-16T19:56:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dd9e4640 1 -- 172.21.15.32:0/562286858 learned_addr learned my addr 172.21.15.32:0/562286858 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:56:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62dd1e3640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d8147840 0x7f62d814c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.544+0000 7f62de1e5640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d814c970 0x7f62d8148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 -- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d8147840 msgr2=0x7f62d814c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d8147840 0x7f62d814c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 -- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d814c970 msgr2=0x7f62d8148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:56:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d814c970 0x7f62d8148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:56:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 -- 172.21.15.32:0/562286858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d814c650 con 0x7f62d8147460 2024-03-16T19:56:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.548+0000 7f62dd9e4640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009fd0 tx=0x7f62d400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.552+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400eea0 con 0x7f62d8147460 2024-03-16T19:56:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.552+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d400c940 con 0x7f62d8147460 2024-03-16T19:56:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.552+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d81eff60 con 0x7f62d8147460 2024-03-16T19:56:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.552+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400ccf0 con 0x7f62d8147460 2024-03-16T19:56:59.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.552+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d81f0520 con 0x7f62d8147460 2024-03-16T19:56:59.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.556+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d8068000 con 0x7f62d8147460 2024-03-16T19:56:59.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.844+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f62d4013470 con 0x7f62d8147460 2024-03-16T19:56:59.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.844+0000 7f62ce7fc640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 0x7f62bc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:56:59.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.844+0000 7f62dd1e3640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 0x7f62bc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:56:59.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.844+0000 7f62dd1e3640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 0x7f62bc069380 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f62d80c9b00 tx=0x7f62c8009320 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:56:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.880+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f62d4055e40 con 0x7f62d8147460 2024-03-16T19:56:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:56:59.880+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f62d40520e0 con 0x7f62d8147460 2024-03-16T19:57:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.056+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62d80722b0 con 0x7f62d8147460 2024-03-16T19:57:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.144+0000 7f62ce7fc640 1 -- 172.21.15.32:0/562286858 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f62d4021030 con 0x7f62d8147460 2024-03-16T19:57:00.117 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:00.117 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 msgr2=0x7f62bc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 0x7f62bc069380 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f62d80c9b00 tx=0x7f62c8009320 comp rx=0 tx=0).stop 2024-03-16T19:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 msgr2=0x7f62d81ef840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009fd0 tx=0x7f62d400ec50 comp rx=0 tx=0).stop 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 shutdown_connections 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f62bc066ec0 0x7f62bc069380 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62d814c970 0x7f62d8148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f62d8147840 0x7f62d814c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 --2- 172.21.15.32:0/562286858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62d8147460 0x7f62d81ef840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 >> 172.21.15.32:0/562286858 conn(0x7f62d806c0d0 msgr2=0x7f62d814fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 shutdown_connections 2024-03-16T19:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.148+0000 7f62dfc48640 1 -- 172.21.15.32:0/562286858 wait complete. 2024-03-16T19:57:00.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:56:31.637719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:56:33.839961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T19:56:40.663264+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T19:57:00.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:57:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.284+0000 7fb6e4a96640 1 Processor -- start 2024-03-16T19:57:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.284+0000 7fb6e4a96640 1 -- start start 2024-03-16T19:57:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.288+0000 7fb6e4a96640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6e00c6fa0 0x7fb6e0163790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.288+0000 7fb6e4a96640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e01526a0 con 0x7fb6e00b15d0 2024-03-16T19:57:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.288+0000 7fb6e4a96640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e0074580 con 0x7fb6e00c7a40 2024-03-16T19:57:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.288+0000 7fb6e4a96640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e00c8360 con 0x7fb6e00c6fa0 2024-03-16T19:57:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.292+0000 7fb6de59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb6e00c7a40 0x7fb6e015b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:38242/0 (socket says 172.21.15.32:38242) 2024-03-16T19:57:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.292+0000 7fb6de59c640 1 -- 172.21.15.32:0/748327207 learned_addr learned my addr 172.21.15.32:0/748327207 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.292+0000 7fb6ded9d640 1 --2- 172.21.15.32:0/748327207 >> v2:172.21.15.59:3300/0 conn(0x7fb6e00c6fa0 0x7fb6e0163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.348+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125026586 0 0) 0x7fb6e01526a0 con 0x7fb6e00b15d0 2024-03-16T19:57:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.348+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003530 con 0x7fb6e00b15d0 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2929730171 0 0) 0x7fb6c8003530 con 0x7fb6e00b15d0 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 >> v2:172.21.15.59:3300/0 conn(0x7fb6e00c6fa0 msgr2=0x7fb6e0163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 --2- 172.21.15.32:0/748327207 >> v2:172.21.15.59:3300/0 conn(0x7fb6e00c6fa0 0x7fb6e0163790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 >> v1:172.21.15.59:6789/0 conn(0x7fb6e00c7a40 legacy=0x7fb6e015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e015bf80 con 0x7fb6e00b15d0 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fb6d0001680 con 0x7fb6e00b15d0 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 >> v1:172.21.15.32:6789/0 conn(0x7fb6e00b15d0 legacy=0x7fb6e00c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6c8003a00 0x7fb6c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6c80047d0 0x7fb6c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6c800fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00c8360 con 0x7fb6c8005190 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c80104d0 con 0x7fb6c80047d0 2024-03-16T19:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.352+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8010690 con 0x7fb6c8003a00 2024-03-16T19:57:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.356+0000 7fb6ded9d640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6c800fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.360+0000 7fb6ded9d640 1 -- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6c8003a00 msgr2=0x7fb6c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.360+0000 7fb6ded9d640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6c8003a00 0x7fb6c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.360+0000 7fb6ded9d640 1 -- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6c80047d0 msgr2=0x7fb6c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.360+0000 7fb6ded9d640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6c80047d0 0x7fb6c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.364+0000 7fb6ded9d640 1 -- 172.21.15.32:0/748327207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8003530 con 0x7fb6c8005190 2024-03-16T19:57:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.368+0000 7fb6ded9d640 1 -- 172.21.15.32:0/748327207 reap_dead start 2024-03-16T19:57:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.368+0000 7fb6ded9d640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6c800fda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009fc0 tx=0x7fb6d801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79445da409afc20a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:00.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80a5cd0 con 0x7fb6c8005190 2024-03-16T19:57:00.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d800f040 con 0x7fb6c8005190 2024-03-16T19:57:00.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6dd59a640 1 -- 172.21.15.32:0/748327207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8003be0 con 0x7fb6c8005190 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 msgr2=0x7fb6c800fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6c800fda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009fc0 tx=0x7fb6d801ccc0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- 172.21.15.32:0/748327207 shutdown_connections 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/748327207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6c800fda0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- 172.21.15.32:0/748327207 >> 172.21.15.32:0/748327207 conn(0x7fb6e006f540 msgr2=0x7fb6e006f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- 172.21.15.32:0/748327207 shutdown_connections 2024-03-16T19:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- 172.21.15.32:0/748327207 wait complete. 2024-03-16T19:57:00.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 Processor -- start 2024-03-16T19:57:00.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.376+0000 7fb6e4a96640 1 -- start start 2024-03-16T19:57:00.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6e014aa90 0x7fb6e01eed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6e01e2c70 0x7fb6e01ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0148e70 con 0x7fb6c8005190 2024-03-16T19:57:00.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e014ff00 con 0x7fb6e014aa90 2024-03-16T19:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6e4a96640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0149980 con 0x7fb6e01e2c70 2024-03-16T19:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54882/0 (socket says 172.21.15.32:54882) 2024-03-16T19:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.380+0000 7fb6de59c640 1 -- 172.21.15.32:0/1662302280 learned_addr learned my addr 172.21.15.32:0/1662302280 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 -- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6e01e2c70 msgr2=0x7fb6e01ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6e01e2c70 0x7fb6e01ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 -- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6e014aa90 msgr2=0x7fb6e01eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6e014aa90 0x7fb6e01eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 -- 172.21.15.32:0/1662302280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e0150d60 con 0x7fb6c8005190 2024-03-16T19:57:00.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.384+0000 7fb6de59c640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c400 tx=0x7fb6d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:00.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4005cf0 con 0x7fb6c8005190 2024-03-16T19:57:00.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d4005e50 con 0x7fb6c8005190 2024-03-16T19:57:00.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d40072f0 con 0x7fb6c8005190 2024-03-16T19:57:00.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e006c730 con 0x7fb6c8005190 2024-03-16T19:57:00.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e0149330 con 0x7fb6c8005190 2024-03-16T19:57:00.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.392+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e006a340 con 0x7fb6c8005190 2024-03-16T19:57:00.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.676+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb6d4007450 con 0x7fb6c8005190 2024-03-16T19:57:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.676+0000 7fb6ceffd640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 0x7fb6c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.676+0000 7fb6ddd9b640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 0x7fb6c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.676+0000 7fb6ddd9b640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 0x7fb6c0069350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0150850 tx=0x7fb6d0001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.704+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb6d4055730 con 0x7fb6c8005190 2024-03-16T19:57:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.704+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb6d40512d0 con 0x7fb6c8005190 2024-03-16T19:57:00.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.896+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6e0071ac0 con 0x7fb6c8005190 2024-03-16T19:57:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.908+0000 7fb6ceffd640 1 -- 172.21.15.32:0/1662302280 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6d4017020 con 0x7fb6c8005190 2024-03-16T19:57:00.881 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:57:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 msgr2=0x7fb6c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 0x7fb6c0069350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0150850 tx=0x7fb6d0001630 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 msgr2=0x7fb6e01512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c400 tx=0x7fb6d400c8c0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 shutdown_connections 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb6c0066e90 0x7fb6c0069350 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6e01e2c70 0x7fb6e01ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6e014aa90 0x7fb6e01eed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 --2- 172.21.15.32:0/1662302280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c8005190 0x7fb6e01512e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 >> 172.21.15.32:0/1662302280 conn(0x7fb6e006f540 msgr2=0x7fb6e00cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 shutdown_connections 2024-03-16T19:57:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:00.912+0000 7fb6e4a96640 1 -- 172.21.15.32:0/1662302280 wait complete. 2024-03-16T19:57:00.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:57:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 Processor -- start 2024-03-16T19:57:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 -- start start 2024-03-16T19:57:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1970150f00 0x7f19700c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f19700c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700c7e50 con 0x7f19700c7040 2024-03-16T19:57:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1970070850 con 0x7f1970150f00 2024-03-16T19:57:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f1978124640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19701512e0 con 0x7f19700c3540 2024-03-16T19:57:01.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.064+0000 7f19766c1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f19700c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.068+0000 7f19756bf640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f19700c7040 0x7f19700c3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:38252/0 (socket says 172.21.15.32:38252) 2024-03-16T19:57:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.068+0000 7f19756bf640 1 -- 172.21.15.32:0/2521535644 learned_addr learned my addr 172.21.15.32:0/2521535644 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.068+0000 7f1975ec0640 1 --2- 172.21.15.32:0/2521535644 >> v2:172.21.15.59:3300/0 conn(0x7f1970150f00 0x7f19700c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.072+0000 7f19766c1640 1 -- 172.21.15.32:0/2521535644 >> v2:172.21.15.59:3300/0 conn(0x7f1970150f00 msgr2=0x7f19700c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.072+0000 7f19766c1640 1 --2- 172.21.15.32:0/2521535644 >> v2:172.21.15.59:3300/0 conn(0x7f1970150f00 0x7f19700c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.072+0000 7f19766c1640 1 -- 172.21.15.32:0/2521535644 >> v1:172.21.15.59:6789/0 conn(0x7f19700c7040 legacy=0x7f19700c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.072+0000 7f19766c1640 1 -- 172.21.15.32:0/2521535644 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970070290 con 0x7f19700c3540 2024-03-16T19:57:01.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.080+0000 7f19766c1640 1 --2- 172.21.15.32:0/2521535644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f19700c82e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19600097c0 tx=0x7f196001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=960b686cb5be5bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:01.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.088+0000 7f1974ebe640 1 -- 172.21.15.32:0/2521535644 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600ac410 con 0x7f19700c3540 2024-03-16T19:57:01.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.088+0000 7f1974ebe640 1 -- 172.21.15.32:0/2521535644 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600ac570 con 0x7f19700c3540 2024-03-16T19:57:01.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.088+0000 7f1974ebe640 1 -- 172.21.15.32:0/2521535644 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600ac900 con 0x7f19700c3540 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- 172.21.15.32:0/2521535644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 msgr2=0x7f19700c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- 172.21.15.32:0/2521535644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f19700c82e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19600097c0 tx=0x7f196001c4a0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- 172.21.15.32:0/2521535644 shutdown_connections 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- 172.21.15.32:0/2521535644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f19700c82e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- 172.21.15.32:0/2521535644 >> v2:172.21.15.59:3300/0 conn(0x7f1970150f00 0x7f19700c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- 172.21.15.32:0/2521535644 >> 172.21.15.32:0/2521535644 conn(0x7f1970066ea0 msgr2=0x7f19700721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:01.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- 172.21.15.32:0/2521535644 shutdown_connections 2024-03-16T19:57:01.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- 172.21.15.32:0/2521535644 wait complete. 2024-03-16T19:57:01.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 Processor -- start 2024-03-16T19:57:01.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- start start 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f197014a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19700c7040 0x7f1970143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19700cc460 con 0x7f1970150f00 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197014ae40 con 0x7f19700c7040 2024-03-16T19:57:01.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f1978124640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197014cf80 con 0x7f19700c3540 2024-03-16T19:57:01.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.092+0000 7f19766c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.096+0000 7f19766c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54892/0 (socket says 172.21.15.32:54892) 2024-03-16T19:57:01.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.096+0000 7f19766c1640 1 -- 172.21.15.32:0/857322842 learned_addr learned my addr 172.21.15.32:0/857322842 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:01.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.096+0000 7f19756bf640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19700c7040 0x7f1970143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.096+0000 7f1975ec0640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f197014a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.100+0000 7f19766c1640 1 -- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 msgr2=0x7f197014a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.100+0000 7f19766c1640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f197014a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.100+0000 7f19766c1640 1 -- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19700c7040 msgr2=0x7f1970143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.100+0000 7f19766c1640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19700c7040 0x7f1970143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.100+0000 7f19766c1640 1 -- 172.21.15.32:0/857322842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009420 con 0x7f1970150f00 2024-03-16T19:57:01.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.104+0000 7f19766c1640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f196001a0d0 tx=0x7f1960019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:01.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600b5430 con 0x7f1970150f00 2024-03-16T19:57:01.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600b5590 con 0x7f1970150f00 2024-03-16T19:57:01.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600b5940 con 0x7f1970150f00 2024-03-16T19:57:01.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197014abb0 con 0x7f1970150f00 2024-03-16T19:57:01.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197014f8b0 con 0x7f1970150f00 2024-03-16T19:57:01.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.108+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970068000 con 0x7f1970150f00 2024-03-16T19:57:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.140+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f19600b3040 con 0x7f1970150f00 2024-03-16T19:57:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.140+0000 7f196e7fc640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 0x7f195c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.140+0000 7f1975ec0640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 0x7f195c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.140+0000 7f1975ec0640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 0x7f195c069300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1968011340 tx=0x7f196800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.148+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f19600efba0 con 0x7f1970150f00 2024-03-16T19:57:01.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.428+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f19600ebe40 con 0x7f1970150f00 2024-03-16T19:57:01.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.616+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19700709e0 con 0x7f1970150f00 2024-03-16T19:57:01.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.632+0000 7f196e7fc640 1 -- 172.21.15.32:0/857322842 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f19600b1070 con 0x7f1970150f00 2024-03-16T19:57:01.603 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:01.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 msgr2=0x7f195c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 0x7f195c069300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1968011340 tx=0x7f196800b040 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 msgr2=0x7f19700cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f196001a0d0 tx=0x7f1960019240 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 shutdown_connections 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f195c066e40 0x7f195c069300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1970150f00 0x7f19700cb5c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19700c7040 0x7f1970143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 --2- 172.21.15.32:0/857322842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f19700c3540 0x7f197014a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 >> 172.21.15.32:0/857322842 conn(0x7f1970066ea0 msgr2=0x7f1970156060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 shutdown_connections 2024-03-16T19:57:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.636+0000 7f1978124640 1 -- 172.21.15.32:0/857322842 wait complete. 2024-03-16T19:57:01.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:57:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 Processor -- start 2024-03-16T19:57:01.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 -- start start 2024-03-16T19:57:01.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c500b1a90 0x7f9c5015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c50072c90 con 0x7f9c500c54b0 2024-03-16T19:57:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c500c5890 con 0x7f9c500c7560 2024-03-16T19:57:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c55064640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c50072f00 con 0x7f9c500b1a90 2024-03-16T19:57:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c4e59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c500b1a90 0x7f9c5015b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c4ed9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9c500c54b0 0x7f9c500c6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49566/0 (socket says 172.21.15.32:49566) 2024-03-16T19:57:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.768+0000 7f9c4ed9d640 1 -- 172.21.15.32:0/2562400943 learned_addr learned my addr 172.21.15.32:0/2562400943 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:01.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.772+0000 7f9c4e59c640 1 -- 172.21.15.32:0/2562400943 >> v1:172.21.15.59:6789/0 conn(0x7f9c500c7560 legacy=0x7f9c5015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:01.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.772+0000 7f9c4e59c640 1 -- 172.21.15.32:0/2562400943 >> v1:172.21.15.32:6789/0 conn(0x7f9c500c54b0 legacy=0x7f9c500c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.772+0000 7f9c4e59c640 1 -- 172.21.15.32:0/2562400943 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c500731e0 con 0x7f9c500b1a90 2024-03-16T19:57:01.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.780+0000 7f9c4e59c640 1 --2- 172.21.15.32:0/2562400943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015b3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c440097a0 tx=0x7f9c4401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a305b82e8ce9bd5a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c4dd9b640 1 -- 172.21.15.32:0/2562400943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c440a8930 con 0x7f9c500b1a90 2024-03-16T19:57:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c4dd9b640 1 -- 172.21.15.32:0/2562400943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c440a8a90 con 0x7f9c500b1a90 2024-03-16T19:57:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c4dd9b640 1 -- 172.21.15.32:0/2562400943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c440a8e20 con 0x7f9c500b1a90 2024-03-16T19:57:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 -- 172.21.15.32:0/2562400943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 msgr2=0x7f9c5015b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 --2- 172.21.15.32:0/2562400943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015b3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c440097a0 tx=0x7f9c4401ec30 comp rx=0 tx=0).stop 2024-03-16T19:57:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 -- 172.21.15.32:0/2562400943 shutdown_connections 2024-03-16T19:57:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 --2- 172.21.15.32:0/2562400943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015b3b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 -- 172.21.15.32:0/2562400943 >> 172.21.15.32:0/2562400943 conn(0x7f9c5006f3a0 msgr2=0x7f9c5006f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:01.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 -- 172.21.15.32:0/2562400943 shutdown_connections 2024-03-16T19:57:01.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.784+0000 7f9c55064640 1 -- 172.21.15.32:0/2562400943 wait complete. 2024-03-16T19:57:01.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 Processor -- start 2024-03-16T19:57:01.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 -- start start 2024-03-16T19:57:01.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9c500c54b0 0x7f9c500c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c500c7560 0x7f9c500c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:01.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c501e2080 con 0x7f9c500b1a90 2024-03-16T19:57:01.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5015b070 con 0x7f9c500c7560 2024-03-16T19:57:01.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c55064640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50071b30 con 0x7f9c500c54b0 2024-03-16T19:57:01.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c4ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:01.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c4ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54920/0 (socket says 172.21.15.32:54920) 2024-03-16T19:57:01.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.788+0000 7f9c4ed9d640 1 -- 172.21.15.32:0/1850177433 learned_addr learned my addr 172.21.15.32:0/1850177433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.792+0000 7f9c4ed9d640 1 -- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9c500c54b0 msgr2=0x7f9c500c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.792+0000 7f9c4ed9d640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9c500c54b0 0x7f9c500c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.792+0000 7f9c4ed9d640 1 -- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c500c7560 msgr2=0x7f9c500c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.792+0000 7f9c4ed9d640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c500c7560 0x7f9c500c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.792+0000 7f9c4ed9d640 1 -- 172.21.15.32:0/1850177433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440093b0 con 0x7f9c500b1a90 2024-03-16T19:57:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.796+0000 7f9c4ed9d640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c450 tx=0x7f9c4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40005cf0 con 0x7f9c500b1a90 2024-03-16T19:57:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40005e50 con 0x7f9c500b1a90 2024-03-16T19:57:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40007290 con 0x7f9c500b1a90 2024-03-16T19:57:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c501e1120 con 0x7f9c500b1a90 2024-03-16T19:57:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c501e16e0 con 0x7f9c500b1a90 2024-03-16T19:57:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:01.800+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c50068000 con 0x7f9c500b1a90 2024-03-16T19:57:02.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.092+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9c400073f0 con 0x7f9c500b1a90 2024-03-16T19:57:02.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.092+0000 7f9c377fe640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 0x7f9c3c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.092+0000 7f9c4e59c640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 0x7f9c3c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.092+0000 7f9c4e59c640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 0x7f9c3c069380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9c440b2000 tx=0x7f9c44004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.132+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9c400554d0 con 0x7f9c500b1a90 2024-03-16T19:57:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.132+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9c40051070 con 0x7f9c500b1a90 2024-03-16T19:57:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.304+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c500b1870 con 0x7f9c500b1a90 2024-03-16T19:57:02.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.312+0000 7f9c377fe640 1 -- 172.21.15.32:0/1850177433 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c40054730 con 0x7f9c500b1a90 2024-03-16T19:57:02.285 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 msgr2=0x7f9c3c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 0x7f9c3c069380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9c440b2000 tx=0x7f9c44004740 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 msgr2=0x7f9c5015ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c450 tx=0x7f9c4000c910 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 shutdown_connections 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9c3c066ec0 0x7f9c3c069380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c500c7560 0x7f9c500c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9c500c54b0 0x7f9c500c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 --2- 172.21.15.32:0/1850177433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500b1a90 0x7f9c5015ab30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 >> 172.21.15.32:0/1850177433 conn(0x7f9c5006f3a0 msgr2=0x7f9c5014aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:02.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 shutdown_connections 2024-03-16T19:57:02.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.316+0000 7f9c55064640 1 -- 172.21.15.32:0/1850177433 wait complete. 2024-03-16T19:57:02.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T19:57:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 Processor -- start 2024-03-16T19:57:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 -- start start 2024-03-16T19:57:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 0x7f822415ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8224153f20 0x7f82241543d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240cb080 con 0x7f82240cc510 2024-03-16T19:57:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822415d650 con 0x7f8224153f20 2024-03-16T19:57:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f822a9b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822415d930 con 0x7f82240cc8f0 2024-03-16T19:57:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f8223fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82240cc510 0x7f82240ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49570/0 (socket says 172.21.15.32:49570) 2024-03-16T19:57:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f8223fff640 1 -- 172.21.15.32:0/2727510288 learned_addr learned my addr 172.21.15.32:0/2727510288 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f8228f53640 1 --2- 172.21.15.32:0/2727510288 >> v2:172.21.15.59:3300/0 conn(0x7f8224153f20 0x7f82241543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.456+0000 7f82237fe640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 0x7f822415ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.460+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237461689 0 0) 0x7f82240cb080 con 0x7f82240cc510 2024-03-16T19:57:02.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.460+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f820c003530 con 0x7f82240cc510 2024-03-16T19:57:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872926146 0 0) 0x7f820c003530 con 0x7f82240cc510 2024-03-16T19:57:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 msgr2=0x7f822415ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 0x7f822415ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 >> v2:172.21.15.59:3300/0 conn(0x7f8224153f20 msgr2=0x7f82241543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 --2- 172.21.15.32:0/2727510288 >> v2:172.21.15.59:3300/0 conn(0x7f8224153f20 0x7f82241543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822415dc10 con 0x7f82240cc510 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f8214001680 con 0x7f82240cc510 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 >> v1:172.21.15.32:6789/0 conn(0x7f82240cc510 legacy=0x7f82240ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c003980 0x7f820c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0047d0 0x7f820c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 0x7f820c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822415d650 con 0x7f820c003980 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822415d930 con 0x7f820c0047d0 2024-03-16T19:57:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.464+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c00b9c0 con 0x7f82240cc8f0 2024-03-16T19:57:02.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.468+0000 7f8223fff640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c003980 0x7f820c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 -- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 msgr2=0x7f820c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82240cc8f0 0x7f820c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 -- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0047d0 msgr2=0x7f820c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0047d0 0x7f820c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 -- 172.21.15.32:0/2727510288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c003530 con 0x7f820c003980 2024-03-16T19:57:02.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.472+0000 7f8223fff640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c003980 0x7f820c003da0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f821400c160 tx=0x7f821401f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8575539b9db511e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140a8c30 con 0x7f820c003980 2024-03-16T19:57:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82140a8d90 con 0x7f820c003980 2024-03-16T19:57:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f8222ffd640 1 -- 172.21.15.32:0/2727510288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214005b20 con 0x7f820c003980 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 -- 172.21.15.32:0/2727510288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c003980 msgr2=0x7f820c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 --2- 172.21.15.32:0/2727510288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c003980 0x7f820c003da0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f821400c160 tx=0x7f821401f310 comp rx=0 tx=0).stop 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f8228f53640 1 -- 172.21.15.32:0/2727510288 reap_dead start 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 -- 172.21.15.32:0/2727510288 shutdown_connections 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 -- 172.21.15.32:0/2727510288 >> 172.21.15.32:0/2727510288 conn(0x7f822406cea0 msgr2=0x7f822406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 -- 172.21.15.32:0/2727510288 shutdown_connections 2024-03-16T19:57:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.480+0000 7f822a9b6640 1 -- 172.21.15.32:0/2727510288 wait complete. 2024-03-16T19:57:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 Processor -- start 2024-03-16T19:57:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 -- start start 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c003980 0x7f82241f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f822415d080 0x7f82240c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82241f4260 con 0x7f82240b4b00 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82241f39c0 con 0x7f820c003980 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f822a9b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82241f3ca0 con 0x7f822415d080 2024-03-16T19:57:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f82237fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f82237fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54940/0 (socket says 172.21.15.32:54940) 2024-03-16T19:57:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f82237fe640 1 -- 172.21.15.32:0/1380508416 learned_addr learned my addr 172.21.15.32:0/1380508416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f8228f53640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f822415d080 0x7f82240c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.484+0000 7f8223fff640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c003980 0x7f82241f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.488+0000 7f82237fe640 1 -- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f822415d080 msgr2=0x7f82240c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.488+0000 7f82237fe640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f822415d080 0x7f82240c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.488+0000 7f82237fe640 1 -- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c003980 msgr2=0x7f82241f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.488+0000 7f82237fe640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c003980 0x7f82241f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.488+0000 7f82237fe640 1 -- 172.21.15.32:0/1380508416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82241f3f80 con 0x7f82240b4b00 2024-03-16T19:57:02.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.504+0000 7f82237fe640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f821800e7c0 tx=0x7f821800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800c6d0 con 0x7f82240b4b00 2024-03-16T19:57:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821800c830 con 0x7f82240b4b00 2024-03-16T19:57:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800cbc0 con 0x7f82240b4b00 2024-03-16T19:57:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82241eff60 con 0x7f82240b4b00 2024-03-16T19:57:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822414e2a0 con 0x7f82240b4b00 2024-03-16T19:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.516+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82241efdc0 con 0x7f82240b4b00 2024-03-16T19:57:02.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.544+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f821800cd20 con 0x7f82240b4b00 2024-03-16T19:57:02.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.544+0000 7f8220ff9640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 0x7f8210069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:02.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.544+0000 7f8223fff640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 0x7f8210069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:02.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.548+0000 7f8223fff640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 0x7f8210069350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8214001060 tx=0x7f8214001940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:02.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.548+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8218055b90 con 0x7f82240b4b00 2024-03-16T19:57:02.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:02.968+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8218055960 con 0x7f82240b4b00 2024-03-16T19:57:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.160+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f82241efdc0 con 0x7f82240b4b00 2024-03-16T19:57:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.172+0000 7f8220ff9640 1 -- 172.21.15.32:0/1380508416 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f82241efdc0 con 0x7f82240b4b00 2024-03-16T19:57:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 msgr2=0x7f8210069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 0x7f8210069350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8214001060 tx=0x7f8214001940 comp rx=0 tx=0).stop 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 msgr2=0x7f82241f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f821800e7c0 tx=0x7f821800ebc0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 shutdown_connections 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8210066e90 0x7f8210069350 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f822415d080 0x7f82240c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82240b4b00 0x7f82241f0760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 --2- 172.21.15.32:0/1380508416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c003980 0x7f82241f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 >> 172.21.15.32:0/1380508416 conn(0x7f822406cea0 msgr2=0x7f8224151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 shutdown_connections 2024-03-16T19:57:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.176+0000 7f822a9b6640 1 -- 172.21.15.32:0/1380508416 wait complete. 2024-03-16T19:57:03.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T19:57:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.308+0000 7fc80d2c1640 1 Processor -- start 2024-03-16T19:57:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 -- start start 2024-03-16T19:57:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc808151cc0 0x7fc8080c9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc808067320 con 0x7fc8080c7040 2024-03-16T19:57:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80806bbc0 con 0x7fc808151cc0 2024-03-16T19:57:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc80d2c1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc808157c20 con 0x7fc808158780 2024-03-16T19:57:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:37758/0 (socket says 172.21.15.32:37758) 2024-03-16T19:57:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.312+0000 7fc8077fe640 1 -- 172.21.15.32:0/591064830 learned_addr learned my addr 172.21.15.32:0/591064830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:03.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.316+0000 7fc806ffd640 1 --2- 172.21.15.32:0/591064830 >> v2:172.21.15.59:3300/0 conn(0x7fc808151cc0 0x7fc8080c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.320+0000 7fc8077fe640 1 -- 172.21.15.32:0/591064830 >> v2:172.21.15.59:3300/0 conn(0x7fc808151cc0 msgr2=0x7fc8080c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:03.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.320+0000 7fc8077fe640 1 --2- 172.21.15.32:0/591064830 >> v2:172.21.15.59:3300/0 conn(0x7fc808151cc0 0x7fc8080c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.320+0000 7fc8077fe640 1 -- 172.21.15.32:0/591064830 >> v1:172.21.15.59:6789/0 conn(0x7fc8080c7040 legacy=0x7fc808158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.320+0000 7fc8077fe640 1 -- 172.21.15.32:0/591064830 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080704d0 con 0x7fc808158780 2024-03-16T19:57:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.324+0000 7fc8077fe640 1 --2- 172.21.15.32:0/591064830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc800009e50 tx=0x7fc80001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e82697d4dae7a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:03.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc805ffb640 1 -- 172.21.15.32:0/591064830 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5830 con 0x7fc808158780 2024-03-16T19:57:03.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc805ffb640 1 -- 172.21.15.32:0/591064830 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000a5990 con 0x7fc808158780 2024-03-16T19:57:03.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc805ffb640 1 -- 172.21.15.32:0/591064830 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5d20 con 0x7fc808158780 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- 172.21.15.32:0/591064830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 msgr2=0x7fc8080c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/591064830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc800009e50 tx=0x7fc80001c970 comp rx=0 tx=0).stop 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- 172.21.15.32:0/591064830 shutdown_connections 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/591064830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8080c74b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/591064830 >> v2:172.21.15.59:3300/0 conn(0x7fc808151cc0 0x7fc8080c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- 172.21.15.32:0/591064830 >> 172.21.15.32:0/591064830 conn(0x7fc808066ea0 msgr2=0x7fc8080721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- 172.21.15.32:0/591064830 shutdown_connections 2024-03-16T19:57:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- 172.21.15.32:0/591064830 wait complete. 2024-03-16T19:57:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 Processor -- start 2024-03-16T19:57:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.336+0000 7fc80d2c1640 1 -- start start 2024-03-16T19:57:03.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8080c7040 0x7fc808146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8081f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8080ca890 con 0x7fc808151cc0 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8081f5de0 con 0x7fc8080c7040 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc80d2c1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80814d270 con 0x7fc808158780 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc8067fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc8067fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54950/0 (socket says 172.21.15.32:54950) 2024-03-16T19:57:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc8067fc640 1 -- 172.21.15.32:0/3719072382 learned_addr learned my addr 172.21.15.32:0/3719072382 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc806ffd640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8080c7040 0x7fc808146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.340+0000 7fc8077fe640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8081f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 -- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 msgr2=0x7fc8081f67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8081f67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 -- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8080c7040 msgr2=0x7fc808146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8080c7040 0x7fc808146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 -- 172.21.15.32:0/3719072382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc800009b30 con 0x7fc808151cc0 2024-03-16T19:57:03.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.344+0000 7fc8067fc640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c3c0 tx=0x7fc7fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.348+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc0052d0 con 0x7fc808151cc0 2024-03-16T19:57:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.348+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc005430 con 0x7fc808151cc0 2024-03-16T19:57:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.348+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc003840 con 0x7fc808151cc0 2024-03-16T19:57:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.348+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80814d530 con 0x7fc808151cc0 2024-03-16T19:57:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.348+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80814daf0 con 0x7fc808151cc0 2024-03-16T19:57:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.352+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8081f5f90 con 0x7fc808151cc0 2024-03-16T19:57:03.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.632+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc7fc0039a0 con 0x7fc808151cc0 2024-03-16T19:57:03.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.632+0000 7fc7ef7fe640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 0x7fc7f4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:03.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.632+0000 7fc806ffd640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 0x7fc7f4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:03.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.636+0000 7fc806ffd640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 0x7fc7f4069380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8007a60 tx=0x7fc7f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:03.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.660+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc7fc054cd0 con 0x7fc808151cc0 2024-03-16T19:57:03.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.660+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7fc050f70 con 0x7fc808151cc0 2024-03-16T19:57:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.844+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc808068000 con 0x7fc808151cc0 2024-03-16T19:57:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc7ef7fe640 1 -- 172.21.15.32:0/3719072382 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc7fc054630 con 0x7fc808151cc0 2024-03-16T19:57:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 msgr2=0x7fc7f4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 0x7fc7f4069380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8007a60 tx=0x7fc7f8008040 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 msgr2=0x7fc8081f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c3c0 tx=0x7fc7fc00c880 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 shutdown_connections 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7f4066ec0 0x7fc7f4069380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc808158780 0x7fc8081f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc808151cc0 0x7fc8081f6270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 --2- 172.21.15.32:0/3719072382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8080c7040 0x7fc808146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:03.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 >> 172.21.15.32:0/3719072382 conn(0x7fc808066ea0 msgr2=0x7fc808150690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:03.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 shutdown_connections 2024-03-16T19:57:03.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:03.856+0000 7fc80d2c1640 1 -- 172.21.15.32:0/3719072382 wait complete. 2024-03-16T19:57:03.849 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2024-03-16T19:57:03.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:57:03.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-16T19:57:03.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[32742]: fuse finished with error 0 and tester_r 0 2024-03-16T19:57:03.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T19:57:09.973 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T19:57:09.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:57:09.973 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-16T19:57:09.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 Processor -- start 2024-03-16T19:57:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 -- start start 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f1994073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19940725e0 0x7f1994070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19940708a0 0x7f199415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1994072dd0 con 0x7f19940708a0 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199415d790 con 0x7f19940725e0 2024-03-16T19:57:10.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199b0b0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199415da50 con 0x7f1994150d70 2024-03-16T19:57:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199964d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19940708a0 0x7f199415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199964d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19940708a0 0x7f199415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39494/0 (socket says 172.21.15.32:39494) 2024-03-16T19:57:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f199964d640 1 -- 172.21.15.32:0/2153472348 learned_addr learned my addr 172.21.15.32:0/2153472348 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f1998e4c640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f1994073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.184+0000 7f1993fff640 1 --2- 172.21.15.32:0/2153472348 >> v2:172.21.15.59:3300/0 conn(0x7f19940725e0 0x7f1994070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 -- 172.21.15.32:0/2153472348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 msgr2=0x7f1994073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f1994073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 -- 172.21.15.32:0/2153472348 >> v2:172.21.15.59:3300/0 conn(0x7f19940725e0 msgr2=0x7f1994070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 --2- 172.21.15.32:0/2153472348 >> v2:172.21.15.59:3300/0 conn(0x7f19940725e0 0x7f1994070360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 -- 172.21.15.32:0/2153472348 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199415dd30 con 0x7f19940708a0 2024-03-16T19:57:10.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.188+0000 7f199964d640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940708a0 0x7f199415a420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f199406d040 tx=0x7f198c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ed893ce1e69d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f19937fe640 1 -- 172.21.15.32:0/2153472348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0a57e0 con 0x7f19940708a0 2024-03-16T19:57:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f19937fe640 1 -- 172.21.15.32:0/2153472348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0a5940 con 0x7f19940708a0 2024-03-16T19:57:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f19937fe640 1 -- 172.21.15.32:0/2153472348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0a5cd0 con 0x7f19940708a0 2024-03-16T19:57:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- 172.21.15.32:0/2153472348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940708a0 msgr2=0x7f199415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940708a0 0x7f199415a420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f199406d040 tx=0x7f198c01c960 comp rx=0 tx=0).stop 2024-03-16T19:57:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- 172.21.15.32:0/2153472348 shutdown_connections 2024-03-16T19:57:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940708a0 0x7f199415a420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 --2- 172.21.15.32:0/2153472348 >> v2:172.21.15.59:3300/0 conn(0x7f19940725e0 0x7f1994070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 --2- 172.21.15.32:0/2153472348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f1994073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- 172.21.15.32:0/2153472348 >> 172.21.15.32:0/2153472348 conn(0x7f199406c0d0 msgr2=0x7f199406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- 172.21.15.32:0/2153472348 shutdown_connections 2024-03-16T19:57:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- 172.21.15.32:0/2153472348 wait complete. 2024-03-16T19:57:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 Processor -- start 2024-03-16T19:57:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.196+0000 7f199b0b0640 1 -- start start 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19940708a0 0x7f199414adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f19941ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994160d50 con 0x7f19940725e0 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941eb490 con 0x7f19940708a0 2024-03-16T19:57:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199b0b0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199414b6a0 con 0x7f1994150d70 2024-03-16T19:57:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f1993fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f1993fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39500/0 (socket says 172.21.15.32:39500) 2024-03-16T19:57:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f1993fff640 1 -- 172.21.15.32:0/3897688803 learned_addr learned my addr 172.21.15.32:0/3897688803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f199964d640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f19941ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.200+0000 7f1998e4c640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19940708a0 0x7f199414adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 -- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 msgr2=0x7f19941ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f19941ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 -- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19940708a0 msgr2=0x7f199414adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19940708a0 0x7f199414adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 -- 172.21.15.32:0/3897688803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c009b30 con 0x7f19940725e0 2024-03-16T19:57:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.204+0000 7f1993fff640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f198400d800 tx=0x7f198400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19840096a0 con 0x7f19940725e0 2024-03-16T19:57:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984009800 con 0x7f19940725e0 2024-03-16T19:57:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984009b70 con 0x7f19940725e0 2024-03-16T19:57:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199415d560 con 0x7f19940725e0 2024-03-16T19:57:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199414bc60 con 0x7f19940725e0 2024-03-16T19:57:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.212+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199414be80 con 0x7f19940725e0 2024-03-16T19:57:10.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.236+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1984009cd0 con 0x7f19940725e0 2024-03-16T19:57:10.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.236+0000 7f19917fa640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 0x7f1980069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.236+0000 7f1998e4c640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 0x7f1980069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.236+0000 7f1998e4c640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 0x7f1980069350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f19880079c0 tx=0x7f1988008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.240+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1984056180 con 0x7f19940725e0 2024-03-16T19:57:10.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.520+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1984052420 con 0x7f19940725e0 2024-03-16T19:57:10.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.688+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1994068000 con 0x7f19940725e0 2024-03-16T19:57:10.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.752+0000 7f19917fa640 1 -- 172.21.15.32:0/3897688803 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f1984055ae0 con 0x7f19940725e0 2024-03-16T19:57:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:10.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 msgr2=0x7f1980069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 0x7f1980069350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f19880079c0 tx=0x7f1988008040 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 msgr2=0x7f199414e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f198400d800 tx=0x7f198400dcc0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 shutdown_connections 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1980066e90 0x7f1980069350 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1994150d70 0x7f19941ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940725e0 0x7f199414e5d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 --2- 172.21.15.32:0/3897688803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19940708a0 0x7f199414adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 >> 172.21.15.32:0/3897688803 conn(0x7f199406c0d0 msgr2=0x7f19940c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 shutdown_connections 2024-03-16T19:57:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.756+0000 7f199b0b0640 1 -- 172.21.15.32:0/3897688803 wait complete. 2024-03-16T19:57:10.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.884+0000 7f92175e9640 1 Processor -- start 2024-03-16T19:57:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.884+0000 7f92175e9640 1 -- start start 2024-03-16T19:57:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f92175e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f92100cdf60 0x7f9210151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f92175e9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 0x7f9210152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f92175e9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9210157eb0 con 0x7f92100cdb80 2024-03-16T19:57:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f92175e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100c4c80 con 0x7f92100cdf60 2024-03-16T19:57:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f92175e9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921006fa80 con 0x7f92100c7830 2024-03-16T19:57:10.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f9215385640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f92100cdb80 0x7f921006f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42368/0 (socket says 172.21.15.32:42368) 2024-03-16T19:57:10.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.888+0000 7f9215385640 1 -- 172.21.15.32:0/2871519925 learned_addr learned my addr 172.21.15.32:0/2871519925 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.892+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424336354 0 0) 0x7f9210157eb0 con 0x7f92100cdb80 2024-03-16T19:57:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.892+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc003530 con 0x7f92100cdb80 2024-03-16T19:57:10.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2631962227 0 0) 0x7f91fc003530 con 0x7f92100cdb80 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 msgr2=0x7f9210152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 0x7f9210152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 >> v2:172.21.15.59:3300/0 conn(0x7f92100cdf60 msgr2=0x7f9210151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 --2- 172.21.15.32:0/2871519925 >> v2:172.21.15.59:3300/0 conn(0x7f92100cdf60 0x7f9210151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100c4f40 con 0x7f92100cdb80 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f920c001680 con 0x7f92100cdb80 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 >> v1:172.21.15.32:6789/0 conn(0x7f92100cdb80 legacy=0x7f921006f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91fc003980 0x7f91fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91fc0047d0 0x7f91fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 0x7f91fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100c4c80 con 0x7f91fc003980 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921006fa80 con 0x7f91fc0047d0 2024-03-16T19:57:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc005580 con 0x7f92100c7830 2024-03-16T19:57:10.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.896+0000 7f9215385640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91fc003980 0x7f91fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.900+0000 7f9215385640 1 -- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 msgr2=0x7f91fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.900+0000 7f9215385640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92100c7830 0x7f91fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.900+0000 7f9215385640 1 -- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91fc0047d0 msgr2=0x7f91fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.900+0000 7f9215385640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91fc0047d0 0x7f91fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.900+0000 7f9215385640 1 -- 172.21.15.32:0/2871519925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc003530 con 0x7f91fc003980 2024-03-16T19:57:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.904+0000 7f9215385640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91fc003980 0x7f91fc003da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c0e0 tx=0x7f920c01cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65e9182bbcf3330 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:10.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c0058e0 con 0x7f91fc003980 2024-03-16T19:57:10.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c005a40 con 0x7f91fc003980 2024-03-16T19:57:10.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f9207fff640 1 -- 172.21.15.32:0/2871519925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c005df0 con 0x7f91fc003980 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 -- 172.21.15.32:0/2871519925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91fc003980 msgr2=0x7f91fc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 --2- 172.21.15.32:0/2871519925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91fc003980 0x7f91fc003da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c0e0 tx=0x7f920c01cb50 comp rx=0 tx=0).stop 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f9215b86640 1 -- 172.21.15.32:0/2871519925 reap_dead start 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 -- 172.21.15.32:0/2871519925 shutdown_connections 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 -- 172.21.15.32:0/2871519925 >> 172.21.15.32:0/2871519925 conn(0x7f9210066ea0 msgr2=0x7f92100672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 -- 172.21.15.32:0/2871519925 shutdown_connections 2024-03-16T19:57:10.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.908+0000 7f92175e9640 1 -- 172.21.15.32:0/2871519925 wait complete. 2024-03-16T19:57:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 Processor -- start 2024-03-16T19:57:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 -- start start 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f91fc003980 0x7f92101f1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92100cb890 0x7f92100cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100cb470 con 0x7f9210145570 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210148f40 con 0x7f92100cb890 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f92175e9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9210156850 con 0x7f91fc003980 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f9214b84640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f9214b84640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39522/0 (socket says 172.21.15.32:39522) 2024-03-16T19:57:10.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.912+0000 7f9214b84640 1 -- 172.21.15.32:0/3881413721 learned_addr learned my addr 172.21.15.32:0/3881413721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.916+0000 7f9214b84640 1 -- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f91fc003980 msgr2=0x7f92101f1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.916+0000 7f9214b84640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f91fc003980 0x7f92101f1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.916+0000 7f9214b84640 1 -- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92100cb890 msgr2=0x7f92100cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.916+0000 7f9214b84640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92100cb890 0x7f92100cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.916+0000 7f9214b84640 1 -- 172.21.15.32:0/3881413721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9210156d80 con 0x7f9210145570 2024-03-16T19:57:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.920+0000 7f9214b84640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f920000b550 tx=0x7f920000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200010890 con 0x7f9210145570 2024-03-16T19:57:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92000109f0 con 0x7f9210145570 2024-03-16T19:57:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200010da0 con 0x7f9210145570 2024-03-16T19:57:10.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92100cc250 con 0x7f9210145570 2024-03-16T19:57:10.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9210156aa0 con 0x7f9210145570 2024-03-16T19:57:10.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:10.924+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9210068000 con 0x7f9210145570 2024-03-16T19:57:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.184+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9200023480 con 0x7f9210145570 2024-03-16T19:57:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.184+0000 7f9205ffb640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 0x7f91f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.184+0000 7f9215385640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 0x7f91f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.184+0000 7f9215385640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 0x7f91f4069300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f920c0aa000 tx=0x7f921006ede0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.216+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9200056160 con 0x7f9210145570 2024-03-16T19:57:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.216+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9200055830 con 0x7f9210145570 2024-03-16T19:57:11.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.436+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9210155d90 con 0x7f9210145570 2024-03-16T19:57:11.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.528+0000 7f9205ffb640 1 -- 172.21.15.32:0/3881413721 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f920001e020 con 0x7f9210145570 2024-03-16T19:57:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 msgr2=0x7f91f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 0x7f91f4069300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f920c0aa000 tx=0x7f921006ede0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 msgr2=0x7f921014acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f920000b550 tx=0x7f920000ba10 comp rx=0 tx=0).stop 2024-03-16T19:57:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 shutdown_connections 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f91f4066e40 0x7f91f4069300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92100cb890 0x7f92100cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9210145570 0x7f921014acc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 --2- 172.21.15.32:0/3881413721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f91fc003980 0x7f92101f1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 >> 172.21.15.32:0/3881413721 conn(0x7f9210066ea0 msgr2=0x7f92100c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 shutdown_connections 2024-03-16T19:57:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.532+0000 7f92175e9640 1 -- 172.21.15.32:0/3881413721 wait complete. 2024-03-16T19:57:11.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T19:57:11.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:57:11.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:57:11.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 Processor -- start 2024-03-16T19:57:11.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 -- start start 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac1541a0 0x7f71ac06b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71ac06ba90 0x7f71ac15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71ac15a7c0 0x7f71ac0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac159e30 con 0x7f71ac06ba90 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac15ac50 con 0x7f71ac15a7c0 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.668+0000 7f71b0d19640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac154580 con 0x7f71ac1541a0 2024-03-16T19:57:11.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.672+0000 7f71a9d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71ac06ba90 0x7f71ac15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.672+0000 7f71a9d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71ac06ba90 0x7f71ac15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39528/0 (socket says 172.21.15.32:39528) 2024-03-16T19:57:11.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.672+0000 7f71a9d9b640 1 -- 172.21.15.32:0/3230184225 learned_addr learned my addr 172.21.15.32:0/3230184225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:11.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.672+0000 7f71aad9d640 1 --2- 172.21.15.32:0/3230184225 >> v2:172.21.15.59:3300/0 conn(0x7f71ac15a7c0 0x7f71ac0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.672+0000 7f71aa59c640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac1541a0 0x7f71ac06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 -- 172.21.15.32:0/3230184225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac1541a0 msgr2=0x7f71ac06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac1541a0 0x7f71ac06b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 -- 172.21.15.32:0/3230184225 >> v2:172.21.15.59:3300/0 conn(0x7f71ac15a7c0 msgr2=0x7f71ac0cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/3230184225 >> v2:172.21.15.59:3300/0 conn(0x7f71ac15a7c0 0x7f71ac0cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 -- 172.21.15.32:0/3230184225 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac15d5c0 con 0x7f71ac06ba90 2024-03-16T19:57:11.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.676+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac06ba90 0x7f71ac15a280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06f470 tx=0x7f719401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bf35ca0bcc956f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a959a640 1 -- 172.21.15.32:0/3230184225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940a89d0 con 0x7f71ac06ba90 2024-03-16T19:57:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a959a640 1 -- 172.21.15.32:0/3230184225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71940a8b30 con 0x7f71ac06ba90 2024-03-16T19:57:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a959a640 1 -- 172.21.15.32:0/3230184225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940b0440 con 0x7f71ac06ba90 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- 172.21.15.32:0/3230184225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac06ba90 msgr2=0x7f71ac15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac06ba90 0x7f71ac15a280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06f470 tx=0x7f719401ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- 172.21.15.32:0/3230184225 shutdown_connections 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- 172.21.15.32:0/3230184225 >> v2:172.21.15.59:3300/0 conn(0x7f71ac15a7c0 0x7f71ac0cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac06ba90 0x7f71ac15a280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- 172.21.15.32:0/3230184225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac1541a0 0x7f71ac06b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- 172.21.15.32:0/3230184225 >> 172.21.15.32:0/3230184225 conn(0x7f71ac066ea0 msgr2=0x7f71ac0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- 172.21.15.32:0/3230184225 shutdown_connections 2024-03-16T19:57:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- 172.21.15.32:0/3230184225 wait complete. 2024-03-16T19:57:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 Processor -- start 2024-03-16T19:57:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- start start 2024-03-16T19:57:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac06ba90 0x7f71ac1467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac15a7c0 0x7f71ac14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac145b20 con 0x7f71ac1541a0 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac146480 con 0x7f71ac06ba90 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71b0d19640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac146dc0 con 0x7f71ac15a7c0 2024-03-16T19:57:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39544/0 (socket says 172.21.15.32:39544) 2024-03-16T19:57:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.684+0000 7f71a9d9b640 1 -- 172.21.15.32:0/1126109249 learned_addr learned my addr 172.21.15.32:0/1126109249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.688+0000 7f71aa59c640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac06ba90 0x7f71ac1467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.688+0000 7f71aad9d640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac15a7c0 0x7f71ac14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.692+0000 7f71a9d9b640 1 -- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac15a7c0 msgr2=0x7f71ac14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.692+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac15a7c0 0x7f71ac14e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.692+0000 7f71a9d9b640 1 -- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac06ba90 msgr2=0x7f71ac1467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.692+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac06ba90 0x7f71ac1467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.692+0000 7f71a9d9b640 1 -- 172.21.15.32:0/1126109249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71940094d0 con 0x7f71ac1541a0 2024-03-16T19:57:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.696+0000 7f71a9d9b640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f719401d3c0 tx=0x7f719401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940b1c50 con 0x7f71ac1541a0 2024-03-16T19:57:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71940b1db0 con 0x7f71ac1541a0 2024-03-16T19:57:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940b9660 con 0x7f71ac1541a0 2024-03-16T19:57:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac1470a0 con 0x7f71ac1541a0 2024-03-16T19:57:11.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac145740 con 0x7f71ac1541a0 2024-03-16T19:57:11.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.700+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac1452c0 con 0x7f71ac1541a0 2024-03-16T19:57:11.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.980+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f71940b7070 con 0x7f71ac1541a0 2024-03-16T19:57:11.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.980+0000 7f71a2ffd640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 0x7f718c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:11.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.980+0000 7f71aa59c640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 0x7f718c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:11.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.980+0000 7f71aa59c640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 0x7f718c069350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f71a4011360 tx=0x7f71a400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:11.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.992+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f71940f3dc0 con 0x7f71ac1541a0 2024-03-16T19:57:11.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:11.996+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f71940ef960 con 0x7f71ac1541a0 2024-03-16T19:57:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.204+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f71ac1452c0 con 0x7f71ac1541a0 2024-03-16T19:57:12.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.544+0000 7f71a2ffd640 1 -- 172.21.15.32:0/1126109249 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f71ac1452c0 con 0x7f71ac1541a0 2024-03-16T19:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 msgr2=0x7f718c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 0x7f718c069350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f71a4011360 tx=0x7f71a400d040 comp rx=0 tx=0).stop 2024-03-16T19:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 msgr2=0x7f71ac145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f719401d3c0 tx=0x7f719401bab0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 shutdown_connections 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f718c066e90 0x7f718c069350 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f71ac15a7c0 0x7f71ac14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ac1541a0 0x7f71ac145f40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 --2- 172.21.15.32:0/1126109249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac06ba90 0x7f71ac1467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 >> 172.21.15.32:0/1126109249 conn(0x7f71ac066ea0 msgr2=0x7f71ac06fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 shutdown_connections 2024-03-16T19:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.548+0000 7f71b0d19640 1 -- 172.21.15.32:0/1126109249 wait complete. 2024-03-16T19:57:12.536 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-03-16T19:57:12.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-16T19:57:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.724+0000 7f1f90727640 1 Processor -- start 2024-03-16T19:57:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.724+0000 7f1f90727640 1 -- start start 2024-03-16T19:57:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f90727640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f880c55d0 0x7f1f881588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f90727640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f881526a0 con 0x7f1f88067060 2024-03-16T19:57:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f90727640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f880c6930 con 0x7f1f88071970 2024-03-16T19:57:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f90727640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f88072970 con 0x7f1f880c55d0 2024-03-16T19:57:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f8e4c3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f880c55d0 0x7f1f881588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f8ecc4640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1f88067060 0x7f1f880c72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42380/0 (socket says 172.21.15.32:42380) 2024-03-16T19:57:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.728+0000 7f1f8ecc4640 1 -- 172.21.15.32:0/2224929601 learned_addr learned my addr 172.21.15.32:0/2224929601 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.732+0000 7f1f8e4c3640 1 -- 172.21.15.32:0/2224929601 >> v1:172.21.15.59:6789/0 conn(0x7f1f88071970 legacy=0x7f1f88072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.732+0000 7f1f8e4c3640 1 -- 172.21.15.32:0/2224929601 >> v1:172.21.15.32:6789/0 conn(0x7f1f88067060 legacy=0x7f1f880c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.732+0000 7f1f8e4c3640 1 -- 172.21.15.32:0/2224929601 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880734a0 con 0x7f1f880c55d0 2024-03-16T19:57:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.740+0000 7f1f8e4c3640 1 --2- 172.21.15.32:0/2224929601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f881588e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f780097c0 tx=0x7f1f7801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2482740f4f62b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f8d4c1640 1 -- 172.21.15.32:0/2224929601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780ab610 con 0x7f1f880c55d0 2024-03-16T19:57:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f8d4c1640 1 -- 172.21.15.32:0/2224929601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f780ab770 con 0x7f1f880c55d0 2024-03-16T19:57:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f8d4c1640 1 -- 172.21.15.32:0/2224929601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780abb00 con 0x7f1f880c55d0 2024-03-16T19:57:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- 172.21.15.32:0/2224929601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 msgr2=0x7f1f881588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 --2- 172.21.15.32:0/2224929601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f881588e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f780097c0 tx=0x7f1f7801c410 comp rx=0 tx=0).stop 2024-03-16T19:57:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- 172.21.15.32:0/2224929601 shutdown_connections 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 --2- 172.21.15.32:0/2224929601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f881588e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- 172.21.15.32:0/2224929601 >> 172.21.15.32:0/2224929601 conn(0x7f1f8806f3a0 msgr2=0x7f1f8806f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- 172.21.15.32:0/2224929601 shutdown_connections 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- 172.21.15.32:0/2224929601 wait complete. 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 Processor -- start 2024-03-16T19:57:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- start start 2024-03-16T19:57:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f88067060 0x7f1f8813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f88071970 0x7f1f880c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8806b630 con 0x7f1f880c55d0 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8806b300 con 0x7f1f88067060 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.744+0000 7f1f90727640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f880c1db0 con 0x7f1f88071970 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39550/0 (socket says 172.21.15.32:39550) 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 -- 172.21.15.32:0/3508252911 learned_addr learned my addr 172.21.15.32:0/3508252911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8e4c3640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f88067060 0x7f1f8813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 -- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f88071970 msgr2=0x7f1f880c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f88071970 0x7f1f880c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 -- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f88067060 msgr2=0x7f1f8813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.748+0000 7f1f8ecc4640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f88067060 0x7f1f8813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.752+0000 7f1f8ecc4640 1 -- 172.21.15.32:0/3508252911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009420 con 0x7f1f880c55d0 2024-03-16T19:57:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.752+0000 7f1f8ecc4640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000ca80 tx=0x7f1f8000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000ac30 con 0x7f1f880c55d0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8000ad90 con 0x7f1f880c55d0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80012600 con 0x7f1f880c55d0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f880c2090 con 0x7f1f880c55d0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f880c2650 con 0x7f1f880c55d0 2024-03-16T19:57:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.756+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8806b7e0 con 0x7f1f880c55d0 2024-03-16T19:57:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.780+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1f80007070 con 0x7f1f880c55d0 2024-03-16T19:57:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.780+0000 7f1f7effd640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 0x7f1f6c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.780+0000 7f1f8e4c3640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 0x7f1f6c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.784+0000 7f1f8e4c3640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 0x7f1f6c069350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1f780af000 tx=0x7f1f780a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:12.784+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1f800557f0 con 0x7f1f880c55d0 2024-03-16T19:57:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.084+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1f80051a90 con 0x7f1f880c55d0 2024-03-16T19:57:13.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.292+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f1f8806b7e0 con 0x7f1f880c55d0 2024-03-16T19:57:13.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.800+0000 7f1f7effd640 1 -- 172.21.15.32:0/3508252911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1f8806b7e0 con 0x7f1f880c55d0 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 msgr2=0x7f1f6c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 0x7f1f6c069350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1f780af000 tx=0x7f1f780a7000 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 msgr2=0x7f1f88140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000ca80 tx=0x7f1f8000cf40 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 shutdown_connections 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1f6c066e90 0x7f1f6c069350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f880c55d0 0x7f1f88140f10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f88071970 0x7f1f880c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 --2- 172.21.15.32:0/3508252911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f88067060 0x7f1f8813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 >> 172.21.15.32:0/3508252911 conn(0x7f1f8806f3a0 msgr2=0x7f1f88072d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 shutdown_connections 2024-03-16T19:57:13.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.804+0000 7f1f90727640 1 -- 172.21.15.32:0/3508252911 wait complete. 2024-03-16T19:57:13.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-16T19:57:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 Processor -- start 2024-03-16T19:57:13.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 -- start start 2024-03-16T19:57:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85bc0cc5d0 0x7f85bc0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc154580 con 0x7f85bc1541a0 2024-03-16T19:57:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc15d5c0 con 0x7f85bc15a280 2024-03-16T19:57:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.948+0000 7f85c41bd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc159e30 con 0x7f85bc0cc5d0 2024-03-16T19:57:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.952+0000 7f85c1758640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f85bc15a280 0x7f85bc15a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:58112/0 (socket says 172.21.15.32:58112) 2024-03-16T19:57:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:13.952+0000 7f85c1758640 1 -- 172.21.15.32:0/2373428591 learned_addr learned my addr 172.21.15.32:0/2373428591 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:13.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.024+0000 7f85c275a640 1 --2- 172.21.15.32:0/2373428591 >> v2:172.21.15.32:3300/0 conn(0x7f85bc0cc5d0 0x7f85bc0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.032+0000 7f85c275a640 1 -- 172.21.15.32:0/2373428591 >> v1:172.21.15.59:6789/0 conn(0x7f85bc15a280 legacy=0x7f85bc15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.032+0000 7f85c275a640 1 -- 172.21.15.32:0/2373428591 >> v1:172.21.15.32:6789/0 conn(0x7f85bc1541a0 legacy=0x7f85bc06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.032+0000 7f85c275a640 1 -- 172.21.15.32:0/2373428591 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc15d8a0 con 0x7f85bc0cc5d0 2024-03-16T19:57:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.040+0000 7f85c275a640 1 --2- 172.21.15.32:0/2373428591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc0cc5d0 0x7f85bc0cca60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009d90 tx=0x7f85b401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43620ca826d2ec47 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c0f57640 1 -- 172.21.15.32:0/2373428591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40af600 con 0x7f85bc0cc5d0 2024-03-16T19:57:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c0f57640 1 -- 172.21.15.32:0/2373428591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b40ab050 con 0x7f85bc0cc5d0 2024-03-16T19:57:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c0f57640 1 -- 172.21.15.32:0/2373428591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40af980 con 0x7f85bc0cc5d0 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 -- 172.21.15.32:0/2373428591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc0cc5d0 msgr2=0x7f85bc0cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2373428591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc0cc5d0 0x7f85bc0cca60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009d90 tx=0x7f85b401f150 comp rx=0 tx=0).stop 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 -- 172.21.15.32:0/2373428591 shutdown_connections 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2373428591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc0cc5d0 0x7f85bc0cca60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 -- 172.21.15.32:0/2373428591 >> 172.21.15.32:0/2373428591 conn(0x7f85bc066ea0 msgr2=0x7f85bc0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 -- 172.21.15.32:0/2373428591 shutdown_connections 2024-03-16T19:57:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.052+0000 7f85c41bd640 1 -- 172.21.15.32:0/2373428591 wait complete. 2024-03-16T19:57:14.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 Processor -- start 2024-03-16T19:57:14.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 -- start start 2024-03-16T19:57:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc0cc5d0 0x7f85bc15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc15a280 0x7f85bc146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1ef7d0 con 0x7f85bc1541a0 2024-03-16T19:57:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1f0010 con 0x7f85bc15a280 2024-03-16T19:57:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.056+0000 7f85c41bd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc156810 con 0x7f85bc0cc5d0 2024-03-16T19:57:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.060+0000 7f85c1758640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.060+0000 7f85c1758640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39570/0 (socket says 172.21.15.32:39570) 2024-03-16T19:57:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.060+0000 7f85c1758640 1 -- 172.21.15.32:0/2198648674 learned_addr learned my addr 172.21.15.32:0/2198648674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.064+0000 7f85c1758640 1 -- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc0cc5d0 msgr2=0x7f85bc15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.064+0000 7f85c1758640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc0cc5d0 0x7f85bc15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.064+0000 7f85c1758640 1 -- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc15a280 msgr2=0x7f85bc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.064+0000 7f85c1758640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc15a280 0x7f85bc146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.064+0000 7f85c1758640 1 -- 172.21.15.32:0/2198648674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4009a70 con 0x7f85bc1541a0 2024-03-16T19:57:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.072+0000 7f85c1758640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c3a0 tx=0x7f85ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.076+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac005cf0 con 0x7f85bc1541a0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.076+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac005e50 con 0x7f85bc1541a0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.076+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc15cc30 con 0x7f85bc1541a0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.076+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0072f0 con 0x7f85bc1541a0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.076+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc1ef9c0 con 0x7f85bc1541a0 2024-03-16T19:57:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.080+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc068000 con 0x7f85bc1541a0 2024-03-16T19:57:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.368+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85ac007450 con 0x7f85bc1541a0 2024-03-16T19:57:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.368+0000 7f85b27fc640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.368+0000 7f85c1f59640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.368+0000 7f85c1f59640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f70 tx=0x7f85b80079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.404+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f85ac055640 con 0x7f85bc1541a0 2024-03-16T19:57:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.404+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f85ac0511e0 con 0x7f85bc1541a0 2024-03-16T19:57:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:14.620+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f85bc072330 con 0x7f85bc1541a0 2024-03-16T19:57:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.260+0000 7f85b27fc640 1 -- 172.21.15.32:0/2198648674 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85ac0548a0 con 0x7f85bc1541a0 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 msgr2=0x7f85a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f70 tx=0x7f85b80079e0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 msgr2=0x7f85bc1562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c3a0 tx=0x7f85ac00c860 comp rx=0 tx=0).stop 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 shutdown_connections 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc15a280 0x7f85bc146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc1541a0 0x7f85bc1562d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 --2- 172.21.15.32:0/2198648674 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc0cc5d0 0x7f85bc15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 >> 172.21.15.32:0/2198648674 conn(0x7f85bc066ea0 msgr2=0x7f85bc0c5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 shutdown_connections 2024-03-16T19:57:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.264+0000 7f85c41bd640 1 -- 172.21.15.32:0/2198648674 wait complete. 2024-03-16T19:57:15.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T19:57:15.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 Processor -- start 2024-03-16T19:57:15.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 -- start start 2024-03-16T19:57:15.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7eff38071930 0x7eff380c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff380c7aa0 0x7eff38159e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff380c6aa0 con 0x7eff380c7aa0 2024-03-16T19:57:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff38073550 con 0x7eff38071930 2024-03-16T19:57:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.404+0000 7eff3d3bf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3815dee0 con 0x7eff380c7180 2024-03-16T19:57:15.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.408+0000 7eff36ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:15.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.408+0000 7eff367fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7eff38071930 0x7eff380c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:15.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.408+0000 7eff36ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:57426/0 (socket says 172.21.15.32:57426) 2024-03-16T19:57:15.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.408+0000 7eff36ffd640 1 -- 172.21.15.32:0/444980441 learned_addr learned my addr 172.21.15.32:0/444980441 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:15.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.412+0000 7eff36ffd640 1 -- 172.21.15.32:0/444980441 >> v2:172.21.15.59:3300/0 conn(0x7eff38071930 msgr2=0x7eff380c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:15.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.412+0000 7eff36ffd640 1 --2- 172.21.15.32:0/444980441 >> v2:172.21.15.59:3300/0 conn(0x7eff38071930 0x7eff380c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.412+0000 7eff36ffd640 1 -- 172.21.15.32:0/444980441 >> v2:172.21.15.32:3300/0 conn(0x7eff380c7aa0 msgr2=0x7eff38159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:15.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.412+0000 7eff36ffd640 1 --2- 172.21.15.32:0/444980441 >> v2:172.21.15.32:3300/0 conn(0x7eff380c7aa0 0x7eff38159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.412+0000 7eff36ffd640 1 -- 172.21.15.32:0/444980441 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3815e1c0 con 0x7eff380c7180 2024-03-16T19:57:15.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.420+0000 7eff36ffd640 1 --2- 172.21.15.32:0/444980441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff380c9a50 tx=0x7eff2801c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=294559daeb07b7f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:15.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff35ffb640 1 -- 172.21.15.32:0/444980441 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280ab600 con 0x7eff380c7180 2024-03-16T19:57:15.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff35ffb640 1 -- 172.21.15.32:0/444980441 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff280a7050 con 0x7eff380c7180 2024-03-16T19:57:15.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff35ffb640 1 -- 172.21.15.32:0/444980441 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280ab980 con 0x7eff380c7180 2024-03-16T19:57:15.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- 172.21.15.32:0/444980441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 msgr2=0x7eff38072370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:15.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/444980441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff380c9a50 tx=0x7eff2801c3d0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- 172.21.15.32:0/444980441 shutdown_connections 2024-03-16T19:57:15.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/444980441 >> v2:172.21.15.32:3300/0 conn(0x7eff380c7aa0 0x7eff38159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/444980441 >> v2:172.21.15.59:3300/0 conn(0x7eff38071930 0x7eff380c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/444980441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff38072370 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- 172.21.15.32:0/444980441 >> 172.21.15.32:0/444980441 conn(0x7eff3806f3d0 msgr2=0x7eff3806f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- 172.21.15.32:0/444980441 shutdown_connections 2024-03-16T19:57:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- 172.21.15.32:0/444980441 wait complete. 2024-03-16T19:57:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 Processor -- start 2024-03-16T19:57:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- start start 2024-03-16T19:57:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff3814d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff380c7aa0 0x7eff380b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff38148b60 con 0x7eff380c7aa0 2024-03-16T19:57:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380adb70 con 0x7eff38071930 2024-03-16T19:57:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.428+0000 7eff3d3bf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380caa40 con 0x7eff380c7180 2024-03-16T19:57:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.432+0000 7eff36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.432+0000 7eff36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:44040/0 (socket says 172.21.15.32:44040) 2024-03-16T19:57:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.432+0000 7eff36ffd640 1 -- 172.21.15.32:0/2552250578 learned_addr learned my addr 172.21.15.32:0/2552250578 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.432+0000 7eff367fc640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff3814d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.444+0000 7eff36ffd640 1 -- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 msgr2=0x7eff3814d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.444+0000 7eff36ffd640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff3814d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.444+0000 7eff36ffd640 1 -- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff380c7aa0 msgr2=0x7eff380b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.444+0000 7eff36ffd640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff380c7aa0 0x7eff380b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.444+0000 7eff36ffd640 1 -- 172.21.15.32:0/2552250578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280094d0 con 0x7eff38071930 2024-03-16T19:57:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.452+0000 7eff36ffd640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff2801cc70 tx=0x7eff28019170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.464+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280b4420 con 0x7eff38071930 2024-03-16T19:57:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.464+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff280b4580 con 0x7eff38071930 2024-03-16T19:57:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.464+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280b4930 con 0x7eff38071930 2024-03-16T19:57:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.464+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff380cace0 con 0x7eff38071930 2024-03-16T19:57:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.464+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff381f6660 con 0x7eff38071930 2024-03-16T19:57:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.468+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff381f61e0 con 0x7eff38071930 2024-03-16T19:57:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.852+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7eff280b2070 con 0x7eff38071930 2024-03-16T19:57:15.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.856+0000 7eff1f7fe640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 0x7eff24069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:15.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.856+0000 7eff367fc640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 0x7eff24069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:15.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.856+0000 7eff367fc640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 0x7eff24069350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff3806fc00 tx=0x7eff2c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.860+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7eff280eee80 con 0x7eff38071930 2024-03-16T19:57:15.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:15.892+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7eff280eb120 con 0x7eff38071930 2024-03-16T19:57:16.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.108+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7eff381f61e0 con 0x7eff38071930 2024-03-16T19:57:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.600+0000 7eff1f7fe640 1 -- 172.21.15.32:0/2552250578 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 115+0+0 (secure 0 0 0) 0x7eff381f61e0 con 0x7eff38071930 2024-03-16T19:57:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 msgr2=0x7eff24069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 0x7eff24069350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff3806fc00 tx=0x7eff2c009190 comp rx=0 tx=0).stop 2024-03-16T19:57:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 msgr2=0x7eff38148620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff2801cc70 tx=0x7eff28019170 comp rx=0 tx=0).stop 2024-03-16T19:57:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 shutdown_connections 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff24066e90 0x7eff24069350 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff380c7aa0 0x7eff380b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff380c7180 0x7eff3814d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 --2- 172.21.15.32:0/2552250578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff38071930 0x7eff38148620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 >> 172.21.15.32:0/2552250578 conn(0x7eff3806f3d0 msgr2=0x7eff3814de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 shutdown_connections 2024-03-16T19:57:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.604+0000 7eff3d3bf640 1 -- 172.21.15.32:0/2552250578 wait complete. 2024-03-16T19:57:16.596 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-16T19:57:16.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:57:16.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 Processor -- start 2024-03-16T19:57:16.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 -- start start 2024-03-16T19:57:16.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd80015b500 0x7fd80015b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:16.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000c7730 con 0x7fd800154250 2024-03-16T19:57:16.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000c7a10 con 0x7fd80015d600 2024-03-16T19:57:16.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd807274640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8000c7cf0 con 0x7fd80015b500 2024-03-16T19:57:16.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.756+0000 7fd805811640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd80015b500 0x7fd80015b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:16.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.760+0000 7fd80480f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd80015d600 0x7fd8000c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:58114/0 (socket says 172.21.15.32:58114) 2024-03-16T19:57:16.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.760+0000 7fd80480f640 1 -- 172.21.15.32:0/624210693 learned_addr learned my addr 172.21.15.32:0/624210693 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.764+0000 7fd7f7fff640 1 -- 172.21.15.32:0/624210693 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467379999 0 0) 0x7fd8000c7730 con 0x7fd800154250 2024-03-16T19:57:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.764+0000 7fd7f7fff640 1 -- 172.21.15.32:0/624210693 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ec003530 con 0x7fd800154250 2024-03-16T19:57:16.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.768+0000 7fd805811640 1 -- 172.21.15.32:0/624210693 >> v1:172.21.15.59:6789/0 conn(0x7fd80015d600 legacy=0x7fd8000c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.768+0000 7fd805811640 1 -- 172.21.15.32:0/624210693 >> v1:172.21.15.32:6789/0 conn(0x7fd800154250 legacy=0x7fd8000cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.768+0000 7fd805811640 1 -- 172.21.15.32:0/624210693 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8000c7fd0 con 0x7fd80015b500 2024-03-16T19:57:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.776+0000 7fd805811640 1 --2- 172.21.15.32:0/624210693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80015b500 0x7fd80015b990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009d90 tx=0x7fd7f801f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d910d2a65d4894 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:16.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.792+0000 7fd7f7fff640 1 -- 172.21.15.32:0/624210693 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80af600 con 0x7fd80015b500 2024-03-16T19:57:16.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.792+0000 7fd7f7fff640 1 -- 172.21.15.32:0/624210693 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f80ab050 con 0x7fd80015b500 2024-03-16T19:57:16.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.792+0000 7fd7f7fff640 1 -- 172.21.15.32:0/624210693 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80af980 con 0x7fd80015b500 2024-03-16T19:57:16.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- 172.21.15.32:0/624210693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80015b500 msgr2=0x7fd80015b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 --2- 172.21.15.32:0/624210693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80015b500 0x7fd80015b990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009d90 tx=0x7fd7f801f150 comp rx=0 tx=0).stop 2024-03-16T19:57:16.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- 172.21.15.32:0/624210693 shutdown_connections 2024-03-16T19:57:16.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 --2- 172.21.15.32:0/624210693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80015b500 0x7fd80015b990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- 172.21.15.32:0/624210693 >> 172.21.15.32:0/624210693 conn(0x7fd800072cf0 msgr2=0x7fd80006bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:16.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- 172.21.15.32:0/624210693 shutdown_connections 2024-03-16T19:57:16.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- 172.21.15.32:0/624210693 wait complete. 2024-03-16T19:57:16.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 Processor -- start 2024-03-16T19:57:16.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.796+0000 7fd807274640 1 -- start start 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800154250 0x7fd800156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80015b500 0x7fd8000c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800155450 con 0x7fd80015d600 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001ef240 con 0x7fd800154250 2024-03-16T19:57:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd807274640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd800154b20 con 0x7fd80015b500 2024-03-16T19:57:16.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd805811640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:16.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd805811640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39596/0 (socket says 172.21.15.32:39596) 2024-03-16T19:57:16.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.800+0000 7fd805811640 1 -- 172.21.15.32:0/2627178804 learned_addr learned my addr 172.21.15.32:0/2627178804 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.804+0000 7fd805811640 1 -- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80015b500 msgr2=0x7fd8000c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.804+0000 7fd805811640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80015b500 0x7fd8000c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.804+0000 7fd805811640 1 -- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800154250 msgr2=0x7fd800156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.804+0000 7fd805811640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800154250 0x7fd800156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.804+0000 7fd805811640 1 -- 172.21.15.32:0/2627178804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f8009a70 con 0x7fd80015d600 2024-03-16T19:57:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.808+0000 7fd805811640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801cd80 tx=0x7fd7f801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80afc20 con 0x7fd80015d600 2024-03-16T19:57:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f80afd80 con 0x7fd80015d600 2024-03-16T19:57:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80ba620 con 0x7fd80015d600 2024-03-16T19:57:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd800154e00 con 0x7fd80015d600 2024-03-16T19:57:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001574d0 con 0x7fd80015d600 2024-03-16T19:57:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.812+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800154c60 con 0x7fd80015d600 2024-03-16T19:57:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.844+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd7f80ba780 con 0x7fd80015d600 2024-03-16T19:57:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.844+0000 7fd7f5ffb640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 0x7fd7e4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.844+0000 7fd805010640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 0x7fd7e4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.844+0000 7fd805010640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 0x7fd7e4069380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc007fd0 tx=0x7fd7fc007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:16.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:16.848+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7f80f47c0 con 0x7fd80015d600 2024-03-16T19:57:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.136+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd7f80f0360 con 0x7fd80015d600 2024-03-16T19:57:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.340+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd800154c60 con 0x7fd80015d600 2024-03-16T19:57:17.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.360+0000 7fd7f5ffb640 1 -- 172.21.15.32:0/2627178804 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6591 (secure 0 0 0) 0x7fd800154c60 con 0x7fd80015d600 2024-03-16T19:57:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:17.333 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:57:16.534782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4246":{"gid":4246,"name":"b","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4252]},"id":1}]} 2024-03-16T19:57:17.334 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 msgr2=0x7fd7e4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 0x7fd7e4069380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc007fd0 tx=0x7fd7fc007900 comp rx=0 tx=0).stop 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 msgr2=0x7fd8001558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801cd80 tx=0x7fd7f801bef0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 shutdown_connections 2024-03-16T19:57:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7e4066ec0 0x7fd7e4069380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80015d600 0x7fd8001558e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80015b500 0x7fd8000c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 --2- 172.21.15.32:0/2627178804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800154250 0x7fd800156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 >> 172.21.15.32:0/2627178804 conn(0x7fd800072cf0 msgr2=0x7fd8000cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 shutdown_connections 2024-03-16T19:57:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.364+0000 7fd807274640 1 -- 172.21.15.32:0/2627178804 wait complete. 2024-03-16T19:57:17.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 Processor -- start 2024-03-16T19:57:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 -- start start 2024-03-16T19:57:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 0x7f9648151330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964806e6a0 con 0x7f9648073050 2024-03-16T19:57:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964806c250 con 0x7f9648072c70 2024-03-16T19:57:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.508+0000 7f964e975640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9648156f20 con 0x7f96480c16a0 2024-03-16T19:57:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.512+0000 7f96477fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9648073050 0x7f9648150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42418/0 (socket says 172.21.15.32:42418) 2024-03-16T19:57:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.512+0000 7f96477fe640 1 -- 172.21.15.32:0/3248035010 learned_addr learned my addr 172.21.15.32:0/3248035010 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.512+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049531583 0 0) 0x7f964806e6a0 con 0x7f9648073050 2024-03-16T19:57:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.512+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630003530 con 0x7f9648073050 2024-03-16T19:57:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243102718 0 0) 0x7f9630003530 con 0x7f9648073050 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 msgr2=0x7f9648151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 0x7f9648151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 >> v1:172.21.15.59:6789/0 conn(0x7f9648072c70 legacy=0x7f96480c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964806c510 con 0x7f9648073050 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f963c001680 con 0x7f9648073050 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 >> v1:172.21.15.32:6789/0 conn(0x7f9648073050 legacy=0x7f9648150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f9630003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 0x7f96300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9630004e30 0x7f963000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300054b0 con 0x7f96300039b0 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9630005620 con 0x7f9630004e30 2024-03-16T19:57:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000e0e0 con 0x7f96480c16a0 2024-03-16T19:57:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.520+0000 7f9647fff640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f9630003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.524+0000 7f9647fff640 1 -- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 msgr2=0x7f96300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.524+0000 7f9647fff640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480c16a0 0x7f96300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.524+0000 7f9647fff640 1 -- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9630004e30 msgr2=0x7f963000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.524+0000 7f9647fff640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9630004e30 0x7f963000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.524+0000 7f9647fff640 1 -- 172.21.15.32:0/3248035010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630003530 con 0x7f96300039b0 2024-03-16T19:57:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.528+0000 7f9647fff640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f9630003db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9640001d50 tx=0x7f964001c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe3785d1e5a0c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640003ba0 con 0x7f96300039b0 2024-03-16T19:57:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640003d00 con 0x7f96300039b0 2024-03-16T19:57:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f9646ffd640 1 -- 172.21.15.32:0/3248035010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400052b0 con 0x7f96300039b0 2024-03-16T19:57:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- 172.21.15.32:0/3248035010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 msgr2=0x7f9630003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 --2- 172.21.15.32:0/3248035010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f9630003db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9640001d50 tx=0x7f964001c650 comp rx=0 tx=0).stop 2024-03-16T19:57:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964cf12640 1 -- 172.21.15.32:0/3248035010 reap_dead start 2024-03-16T19:57:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- 172.21.15.32:0/3248035010 shutdown_connections 2024-03-16T19:57:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- 172.21.15.32:0/3248035010 >> 172.21.15.32:0/3248035010 conn(0x7f964806afc0 msgr2=0x7f964806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- 172.21.15.32:0/3248035010 shutdown_connections 2024-03-16T19:57:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- 172.21.15.32:0/3248035010 wait complete. 2024-03-16T19:57:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 Processor -- start 2024-03-16T19:57:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.532+0000 7f964e975640 1 -- start start 2024-03-16T19:57:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480b12d0 0x7f964814a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964813eed0 0x7f964813f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648141c20 con 0x7f96300039b0 2024-03-16T19:57:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964813ea40 con 0x7f964813eed0 2024-03-16T19:57:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f964e975640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9648141340 con 0x7f96480b12d0 2024-03-16T19:57:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f9647fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f9647fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39600/0 (socket says 172.21.15.32:39600) 2024-03-16T19:57:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.536+0000 7f9647fff640 1 -- 172.21.15.32:0/2024133731 learned_addr learned my addr 172.21.15.32:0/2024133731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:17.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 -- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480b12d0 msgr2=0x7f964814a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480b12d0 0x7f964814a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 -- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964813eed0 msgr2=0x7f964813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964813eed0 0x7f964813f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 -- 172.21.15.32:0/2024133731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480ca220 con 0x7f96300039b0 2024-03-16T19:57:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.540+0000 7f9647fff640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9640001d50 tx=0x7f9630005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.544+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640003980 con 0x7f96300039b0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.544+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640003ae0 con 0x7f96300039b0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.544+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640006d40 con 0x7f96300039b0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.544+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964813d680 con 0x7f96300039b0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.544+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964813dc40 con 0x7f96300039b0 2024-03-16T19:57:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.548+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96480674e0 con 0x7f96300039b0 2024-03-16T19:57:17.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.808+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f96400a9450 con 0x7f96300039b0 2024-03-16T19:57:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.808+0000 7f9644ff9640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 0x7f9634069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.808+0000 7f96477fe640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 0x7f9634069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.812+0000 7f96477fe640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 0x7f9634069350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f96481499f0 tx=0x7f963c0015d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:17.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.844+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f96400ef580 con 0x7f96300039b0 2024-03-16T19:57:17.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:17.844+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f96400eb820 con 0x7f96300039b0 2024-03-16T19:57:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.028+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9648068000 con 0x7f96300039b0 2024-03-16T19:57:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.116+0000 7f9644ff9640 1 -- 172.21.15.32:0/2024133731 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f96400ae070 con 0x7f96300039b0 2024-03-16T19:57:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:18.090 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 msgr2=0x7f9634069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 0x7f9634069350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f96481499f0 tx=0x7f963c0015d0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 msgr2=0x7f96480c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9640001d50 tx=0x7f9630005800 comp rx=0 tx=0).stop 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 shutdown_connections 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9634066e90 0x7f9634069350 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964813eed0 0x7f964813f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f96480b12d0 0x7f964814a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 --2- 172.21.15.32:0/2024133731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300039b0 0x7f96480c9ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 >> 172.21.15.32:0/2024133731 conn(0x7f964806afc0 msgr2=0x7f964814fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 shutdown_connections 2024-03-16T19:57:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.124+0000 7f964e975640 1 -- 172.21.15.32:0/2024133731 wait complete. 2024-03-16T19:57:18.116 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T19:57:18.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:57:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 Processor -- start 2024-03-16T19:57:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 -- start start 2024-03-16T19:57:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071130 0x7f25b40c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25b4071510 0x7f25b40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b40c2940 con 0x7f25b40c8fe0 2024-03-16T19:57:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b40b2de0 con 0x7f25b4071510 2024-03-16T19:57:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.264+0000 7f25baeed640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40ca1f0 con 0x7f25b4071130 2024-03-16T19:57:18.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.328+0000 7f25b948a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25b40c8fe0 0x7f25b4151b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42424/0 (socket says 172.21.15.32:42424) 2024-03-16T19:57:18.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.328+0000 7f25b948a640 1 -- 172.21.15.32:0/1901306134 learned_addr learned my addr 172.21.15.32:0/1901306134 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.328+0000 7f25b3fff640 1 --2- 172.21.15.32:0/1901306134 >> v2:172.21.15.32:3300/0 conn(0x7f25b4071510 0x7f25b40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.340+0000 7f25b8c89640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071130 0x7f25b40c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.348+0000 7f25b3fff640 1 -- 172.21.15.32:0/1901306134 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071130 msgr2=0x7f25b40c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.348+0000 7f25b3fff640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071130 0x7f25b40c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.348+0000 7f25b3fff640 1 -- 172.21.15.32:0/1901306134 >> v1:172.21.15.32:6789/0 conn(0x7f25b40c8fe0 legacy=0x7f25b4151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.348+0000 7f25b3fff640 1 -- 172.21.15.32:0/1901306134 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b40c2b90 con 0x7f25b4071510 2024-03-16T19:57:18.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.360+0000 7f25b3fff640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b4071510 0x7f25b40c8320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f25a00097c0 tx=0x7f25a001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787ddf0cbd87e499 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25b37fe640 1 -- 172.21.15.32:0/1901306134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00a8990 con 0x7f25b4071510 2024-03-16T19:57:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25b37fe640 1 -- 172.21.15.32:0/1901306134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00a8af0 con 0x7f25b4071510 2024-03-16T19:57:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25b37fe640 1 -- 172.21.15.32:0/1901306134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00a8e80 con 0x7f25b4071510 2024-03-16T19:57:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 -- 172.21.15.32:0/1901306134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b4071510 msgr2=0x7f25b40c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b4071510 0x7f25b40c8320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f25a00097c0 tx=0x7f25a001ec90 comp rx=0 tx=0).stop 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 -- 172.21.15.32:0/1901306134 shutdown_connections 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b4071510 0x7f25b40c8320 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 --2- 172.21.15.32:0/1901306134 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071130 0x7f25b40c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.368+0000 7f25baeed640 1 -- 172.21.15.32:0/1901306134 >> 172.21.15.32:0/1901306134 conn(0x7f25b406b190 msgr2=0x7f25b406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- 172.21.15.32:0/1901306134 shutdown_connections 2024-03-16T19:57:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- 172.21.15.32:0/1901306134 wait complete. 2024-03-16T19:57:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 Processor -- start 2024-03-16T19:57:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- start start 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25b4071130 0x7f25b4153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071510 0x7f25b40cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40af530 con 0x7f25b40c8fe0 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40ce5a0 con 0x7f25b4071130 2024-03-16T19:57:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.372+0000 7f25baeed640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40af8b0 con 0x7f25b4071510 2024-03-16T19:57:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.376+0000 7f25b8c89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25b4071130 0x7f25b4153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.376+0000 7f25b3fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071510 0x7f25b40cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.376+0000 7f25b948a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.376+0000 7f25b948a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39606/0 (socket says 172.21.15.32:39606) 2024-03-16T19:57:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.376+0000 7f25b948a640 1 -- 172.21.15.32:0/1624831155 learned_addr learned my addr 172.21.15.32:0/1624831155 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 -- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071510 msgr2=0x7f25b40cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071510 0x7f25b40cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 -- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25b4071130 msgr2=0x7f25b4153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25b4071130 0x7f25b4153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 -- 172.21.15.32:0/1624831155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009420 con 0x7f25b40c8fe0 2024-03-16T19:57:18.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.380+0000 7f25b948a640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00ca80 tx=0x7f25ac00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00ac30 con 0x7f25b40c8fe0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00ad90 con 0x7f25b40c8fe0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac012600 con 0x7f25b40c8fe0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b40afb90 con 0x7f25b40c8fe0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b40b0150 con 0x7f25b40c8fe0 2024-03-16T19:57:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.388+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b406be80 con 0x7f25b40c8fe0 2024-03-16T19:57:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.416+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25ac007070 con 0x7f25b40c8fe0 2024-03-16T19:57:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.420+0000 7f25b17fa640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 0x7f2598069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.420+0000 7f25b8c89640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 0x7f2598069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:18.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.420+0000 7f25b8c89640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 0x7f2598069300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009b60 tx=0x7f25a8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.424+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f25ac055980 con 0x7f25b40c8fe0 2024-03-16T19:57:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.696+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f25ac051c20 con 0x7f25b40c8fe0 2024-03-16T19:57:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.864+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25b40b08c0 con 0x7f2598066e40 2024-03-16T19:57:18.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.900+0000 7f25b17fa640 1 -- 172.21.15.32:0/1624831155 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332571 (secure 0 0 0) 0x7f25b40b08c0 con 0x7f2598066e40 2024-03-16T19:57:18.874 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:18.876 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T19:57:18.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 msgr2=0x7f2598069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 0x7f2598069300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009b60 tx=0x7f25a8009190 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 msgr2=0x7f25b40ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00ca80 tx=0x7f25ac00cf40 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 shutdown_connections 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2598066e40 0x7f2598069300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b40c8fe0 0x7f25b40ccb50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f25b4071510 0x7f25b40cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 --2- 172.21.15.32:0/1624831155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25b4071130 0x7f25b4153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 >> 172.21.15.32:0/1624831155 conn(0x7f25b406b190 msgr2=0x7f25b4149f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 shutdown_connections 2024-03-16T19:57:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:18.904+0000 7f25baeed640 1 -- 172.21.15.32:0/1624831155 wait complete. 2024-03-16T19:57:18.893 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-16T19:57:18.578660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595630,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":219,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236404,"kb_used_data":3604,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738316,"statfs":{"total":773094113280,"available":772852035584,"internally_reserved":0,"allocated":3690496,"data_stored":2176564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":622,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881323+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T19:56:44.881323+0000","last_peered":"2024-03-16T19:56:44.881323+0000","last_clean":"2024-03-16T19:56:44.881323+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T19:56:44.881323+0000","last_undegraded":"2024-03-16T19:56:44.881323+0000","last_fullsized":"2024-03-16T19:56:44.881323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784985+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T19:56:44.784985+0000","last_peered":"2024-03-16T19:56:44.784985+0000","last_clean":"2024-03-16T19:56:44.784985+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T19:56:44.784985+0000","last_undegraded":"2024-03-16T19:56:44.784985+0000","last_fullsized":"2024-03-16T19:56:44.784985+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782560+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T19:56:44.782560+0000","last_peered":"2024-03-16T19:56:44.782560+0000","last_clean":"2024-03-16T19:56:44.782560+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T19:56:44.782560+0000","last_undegraded":"2024-03-16T19:56:44.782560+0000","last_fullsized":"2024-03-16T19:56:44.782560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785841+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T19:56:44.785841+0000","last_peered":"2024-03-16T19:56:44.785841+0000","last_clean":"2024-03-16T19:56:44.785841+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T19:56:44.785841+0000","last_undegraded":"2024-03-16T19:56:44.785841+0000","last_fullsized":"2024-03-16T19:56:44.785841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880422+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T19:56:44.880422+0000","last_peered":"2024-03-16T19:56:44.880422+0000","last_clean":"2024-03-16T19:56:44.880422+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T19:56:44.880422+0000","last_undegraded":"2024-03-16T19:56:44.880422+0000","last_fullsized":"2024-03-16T19:56:44.880422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868804+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T19:56:44.868804+0000","last_peered":"2024-03-16T19:56:44.868804+0000","last_clean":"2024-03-16T19:56:44.868804+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T19:56:44.868804+0000","last_undegraded":"2024-03-16T19:56:44.868804+0000","last_fullsized":"2024-03-16T19:56:44.868804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885168+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T19:56:44.885168+0000","last_peered":"2024-03-16T19:56:44.885168+0000","last_clean":"2024-03-16T19:56:44.885168+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T19:56:44.885168+0000","last_undegraded":"2024-03-16T19:56:44.885168+0000","last_fullsized":"2024-03-16T19:56:44.885168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883804+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T19:56:44.883804+0000","last_peered":"2024-03-16T19:56:44.883804+0000","last_clean":"2024-03-16T19:56:44.883804+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T19:56:44.883804+0000","last_undegraded":"2024-03-16T19:56:44.883804+0000","last_fullsized":"2024-03-16T19:56:44.883804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869676+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T19:56:44.869676+0000","last_peered":"2024-03-16T19:56:44.869676+0000","last_clean":"2024-03-16T19:56:44.869676+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T19:56:44.869676+0000","last_undegraded":"2024-03-16T19:56:44.869676+0000","last_fullsized":"2024-03-16T19:56:44.869676+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883912+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T19:56:44.883912+0000","last_peered":"2024-03-16T19:56:44.883912+0000","last_clean":"2024-03-16T19:56:44.883912+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T19:56:44.883912+0000","last_undegraded":"2024-03-16T19:56:44.883912+0000","last_fullsized":"2024-03-16T19:56:44.883912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884919+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T19:56:44.884919+0000","last_peered":"2024-03-16T19:56:44.884919+0000","last_clean":"2024-03-16T19:56:44.884919+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T19:56:44.884919+0000","last_undegraded":"2024-03-16T19:56:44.884919+0000","last_fullsized":"2024-03-16T19:56:44.884919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869257+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T19:56:44.869257+0000","last_peered":"2024-03-16T19:56:44.869257+0000","last_clean":"2024-03-16T19:56:44.869257+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T19:56:44.869257+0000","last_undegraded":"2024-03-16T19:56:44.869257+0000","last_fullsized":"2024-03-16T19:56:44.869257+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880296+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T19:56:44.880296+0000","last_peered":"2024-03-16T19:56:44.880296+0000","last_clean":"2024-03-16T19:56:44.880296+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T19:56:44.880296+0000","last_undegraded":"2024-03-16T19:56:44.880296+0000","last_fullsized":"2024-03-16T19:56:44.880296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883063+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T19:56:44.883063+0000","last_peered":"2024-03-16T19:56:44.883063+0000","last_clean":"2024-03-16T19:56:44.883063+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T19:56:44.883063+0000","last_undegraded":"2024-03-16T19:56:44.883063+0000","last_fullsized":"2024-03-16T19:56:44.883063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870662+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T19:56:44.870662+0000","last_peered":"2024-03-16T19:56:44.870662+0000","last_clean":"2024-03-16T19:56:44.870662+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T19:56:44.870662+0000","last_undegraded":"2024-03-16T19:56:44.870662+0000","last_fullsized":"2024-03-16T19:56:44.870662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783555+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T19:56:44.783555+0000","last_peered":"2024-03-16T19:56:44.783555+0000","last_clean":"2024-03-16T19:56:44.783555+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T19:56:44.783555+0000","last_undegraded":"2024-03-16T19:56:44.783555+0000","last_fullsized":"2024-03-16T19:56:44.783555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856445+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T19:56:44.856445+0000","last_peered":"2024-03-16T19:56:44.856445+0000","last_clean":"2024-03-16T19:56:44.856445+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T19:56:44.856445+0000","last_undegraded":"2024-03-16T19:56:44.856445+0000","last_fullsized":"2024-03-16T19:56:44.856445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870742+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T19:56:44.870742+0000","last_peered":"2024-03-16T19:56:44.870742+0000","last_clean":"2024-03-16T19:56:44.870742+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T19:56:44.870742+0000","last_undegraded":"2024-03-16T19:56:44.870742+0000","last_fullsized":"2024-03-16T19:56:44.870742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783732+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T19:56:44.783732+0000","last_peered":"2024-03-16T19:56:44.783732+0000","last_clean":"2024-03-16T19:56:44.783732+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T19:56:44.783732+0000","last_undegraded":"2024-03-16T19:56:44.783732+0000","last_fullsized":"2024-03-16T19:56:44.783732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855768+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T19:56:44.855768+0000","last_peered":"2024-03-16T19:56:44.855768+0000","last_clean":"2024-03-16T19:56:44.855768+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T19:56:44.855768+0000","last_undegraded":"2024-03-16T19:56:44.855768+0000","last_fullsized":"2024-03-16T19:56:44.855768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856952+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T19:56:44.856952+0000","last_peered":"2024-03-16T19:56:44.856952+0000","last_clean":"2024-03-16T19:56:44.856952+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T19:56:44.856952+0000","last_undegraded":"2024-03-16T19:56:44.856952+0000","last_fullsized":"2024-03-16T19:56:44.856952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868777+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T19:56:44.868777+0000","last_peered":"2024-03-16T19:56:44.868777+0000","last_clean":"2024-03-16T19:56:44.868777+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T19:56:44.868777+0000","last_undegraded":"2024-03-16T19:56:44.868777+0000","last_fullsized":"2024-03-16T19:56:44.868777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879259+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T19:56:44.879259+0000","last_peered":"2024-03-16T19:56:44.879259+0000","last_clean":"2024-03-16T19:56:44.879259+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T19:56:44.879259+0000","last_undegraded":"2024-03-16T19:56:44.879259+0000","last_fullsized":"2024-03-16T19:56:44.879259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870617+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T19:56:44.870617+0000","last_peered":"2024-03-16T19:56:44.870617+0000","last_clean":"2024-03-16T19:56:44.870617+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T19:56:44.870617+0000","last_undegraded":"2024-03-16T19:56:44.870617+0000","last_fullsized":"2024-03-16T19:56:44.870617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881303+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T19:56:44.881303+0000","last_peered":"2024-03-16T19:56:44.881303+0000","last_clean":"2024-03-16T19:56:44.881303+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T19:56:44.881303+0000","last_undegraded":"2024-03-16T19:56:44.881303+0000","last_fullsized":"2024-03-16T19:56:44.881303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881781+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T19:56:44.881781+0000","last_peered":"2024-03-16T19:56:44.881781+0000","last_clean":"2024-03-16T19:56:44.881781+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T19:56:44.881781+0000","last_undegraded":"2024-03-16T19:56:44.881781+0000","last_fullsized":"2024-03-16T19:56:44.881781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868298+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T19:56:44.868298+0000","last_peered":"2024-03-16T19:56:44.868298+0000","last_clean":"2024-03-16T19:56:44.868298+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T19:56:44.868298+0000","last_undegraded":"2024-03-16T19:56:44.868298+0000","last_fullsized":"2024-03-16T19:56:44.868298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785469+0000","last_change":"2024-03-16T19:56:36.217377+0000","last_active":"2024-03-16T19:56:44.785469+0000","last_peered":"2024-03-16T19:56:44.785469+0000","last_clean":"2024-03-16T19:56:44.785469+0000","last_became_active":"2024-03-16T19:56:36.216579+0000","last_became_peered":"2024-03-16T19:56:36.216579+0000","last_unstale":"2024-03-16T19:56:44.785469+0000","last_undegraded":"2024-03-16T19:56:44.785469+0000","last_fullsized":"2024-03-16T19:56:44.785469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856847+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T19:56:44.856847+0000","last_peered":"2024-03-16T19:56:44.856847+0000","last_clean":"2024-03-16T19:56:44.856847+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T19:56:44.856847+0000","last_undegraded":"2024-03-16T19:56:44.856847+0000","last_fullsized":"2024-03-16T19:56:44.856847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782416+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T19:56:44.782416+0000","last_peered":"2024-03-16T19:56:44.782416+0000","last_clean":"2024-03-16T19:56:44.782416+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T19:56:44.782416+0000","last_undegraded":"2024-03-16T19:56:44.782416+0000","last_fullsized":"2024-03-16T19:56:44.782416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856282+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T19:56:44.856282+0000","last_peered":"2024-03-16T19:56:44.856282+0000","last_clean":"2024-03-16T19:56:44.856282+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T19:56:44.856282+0000","last_undegraded":"2024-03-16T19:56:44.856282+0000","last_fullsized":"2024-03-16T19:56:44.856282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885652+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T19:56:44.885652+0000","last_peered":"2024-03-16T19:56:44.885652+0000","last_clean":"2024-03-16T19:56:44.885652+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T19:56:44.885652+0000","last_undegraded":"2024-03-16T19:56:44.885652+0000","last_fullsized":"2024-03-16T19:56:44.885652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880177+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T19:56:44.880177+0000","last_peered":"2024-03-16T19:56:44.880177+0000","last_clean":"2024-03-16T19:56:44.880177+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T19:56:44.880177+0000","last_undegraded":"2024-03-16T19:56:44.880177+0000","last_fullsized":"2024-03-16T19:56:44.880177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885381+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T19:56:44.885381+0000","last_peered":"2024-03-16T19:56:44.885381+0000","last_clean":"2024-03-16T19:56:44.885381+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T19:56:44.885381+0000","last_undegraded":"2024-03-16T19:56:44.885381+0000","last_fullsized":"2024-03-16T19:56:44.885381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856509+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T19:56:44.856509+0000","last_peered":"2024-03-16T19:56:44.856509+0000","last_clean":"2024-03-16T19:56:44.856509+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T19:56:44.856509+0000","last_undegraded":"2024-03-16T19:56:44.856509+0000","last_fullsized":"2024-03-16T19:56:44.856509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855979+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T19:56:44.855979+0000","last_peered":"2024-03-16T19:56:44.855979+0000","last_clean":"2024-03-16T19:56:44.855979+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T19:56:44.855979+0000","last_undegraded":"2024-03-16T19:56:44.855979+0000","last_fullsized":"2024-03-16T19:56:44.855979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869818+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T19:56:44.869818+0000","last_peered":"2024-03-16T19:56:44.869818+0000","last_clean":"2024-03-16T19:56:44.869818+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T19:56:44.869818+0000","last_undegraded":"2024-03-16T19:56:44.869818+0000","last_fullsized":"2024-03-16T19:56:44.869818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885665+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T19:56:44.885665+0000","last_peered":"2024-03-16T19:56:44.885665+0000","last_clean":"2024-03-16T19:56:44.885665+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T19:56:44.885665+0000","last_undegraded":"2024-03-16T19:56:44.885665+0000","last_fullsized":"2024-03-16T19:56:44.885665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857231+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T19:56:44.857231+0000","last_peered":"2024-03-16T19:56:44.857231+0000","last_clean":"2024-03-16T19:56:44.857231+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T19:56:44.857231+0000","last_undegraded":"2024-03-16T19:56:44.857231+0000","last_fullsized":"2024-03-16T19:56:44.857231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855744+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T19:56:44.855744+0000","last_peered":"2024-03-16T19:56:44.855744+0000","last_clean":"2024-03-16T19:56:44.855744+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T19:56:44.855744+0000","last_undegraded":"2024-03-16T19:56:44.855744+0000","last_fullsized":"2024-03-16T19:56:44.855744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879840+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T19:56:44.879840+0000","last_peered":"2024-03-16T19:56:44.879840+0000","last_clean":"2024-03-16T19:56:44.879840+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T19:56:44.879840+0000","last_undegraded":"2024-03-16T19:56:44.879840+0000","last_fullsized":"2024-03-16T19:56:44.879840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857201+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T19:56:44.857201+0000","last_peered":"2024-03-16T19:56:44.857201+0000","last_clean":"2024-03-16T19:56:44.857201+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T19:56:44.857201+0000","last_undegraded":"2024-03-16T19:56:44.857201+0000","last_fullsized":"2024-03-16T19:56:44.857201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880083+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T19:56:44.880083+0000","last_peered":"2024-03-16T19:56:44.880083+0000","last_clean":"2024-03-16T19:56:44.880083+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T19:56:44.880083+0000","last_undegraded":"2024-03-16T19:56:44.880083+0000","last_fullsized":"2024-03-16T19:56:44.880083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879559+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T19:56:44.879559+0000","last_peered":"2024-03-16T19:56:44.879559+0000","last_clean":"2024-03-16T19:56:44.879559+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T19:56:44.879559+0000","last_undegraded":"2024-03-16T19:56:44.879559+0000","last_fullsized":"2024-03-16T19:56:44.879559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881219+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T19:56:44.881219+0000","last_peered":"2024-03-16T19:56:44.881219+0000","last_clean":"2024-03-16T19:56:44.881219+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T19:56:44.881219+0000","last_undegraded":"2024-03-16T19:56:44.881219+0000","last_fullsized":"2024-03-16T19:56:44.881219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783051+0000","last_change":"2024-03-16T19:56:36.305013+0000","last_active":"2024-03-16T19:56:44.783051+0000","last_peered":"2024-03-16T19:56:44.783051+0000","last_clean":"2024-03-16T19:56:44.783051+0000","last_became_active":"2024-03-16T19:56:36.303533+0000","last_became_peered":"2024-03-16T19:56:36.303533+0000","last_unstale":"2024-03-16T19:56:44.783051+0000","last_undegraded":"2024-03-16T19:56:44.783051+0000","last_fullsized":"2024-03-16T19:56:44.783051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883237+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T19:56:44.883237+0000","last_peered":"2024-03-16T19:56:44.883237+0000","last_clean":"2024-03-16T19:56:44.883237+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T19:56:44.883237+0000","last_undegraded":"2024-03-16T19:56:44.883237+0000","last_fullsized":"2024-03-16T19:56:44.883237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879868+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T19:56:44.879868+0000","last_peered":"2024-03-16T19:56:44.879868+0000","last_clean":"2024-03-16T19:56:44.879868+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T19:56:44.879868+0000","last_undegraded":"2024-03-16T19:56:44.879868+0000","last_fullsized":"2024-03-16T19:56:44.879868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882315+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T19:56:44.882315+0000","last_peered":"2024-03-16T19:56:44.882315+0000","last_clean":"2024-03-16T19:56:44.882315+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T19:56:44.882315+0000","last_undegraded":"2024-03-16T19:56:44.882315+0000","last_fullsized":"2024-03-16T19:56:44.882315+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884701+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T19:56:44.884701+0000","last_peered":"2024-03-16T19:56:44.884701+0000","last_clean":"2024-03-16T19:56:44.884701+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T19:56:44.884701+0000","last_undegraded":"2024-03-16T19:56:44.884701+0000","last_fullsized":"2024-03-16T19:56:44.884701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881140+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T19:56:44.881140+0000","last_peered":"2024-03-16T19:56:44.881140+0000","last_clean":"2024-03-16T19:56:44.881140+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T19:56:44.881140+0000","last_undegraded":"2024-03-16T19:56:44.881140+0000","last_fullsized":"2024-03-16T19:56:44.881140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880005+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T19:56:44.880005+0000","last_peered":"2024-03-16T19:56:44.880005+0000","last_clean":"2024-03-16T19:56:44.880005+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T19:56:44.880005+0000","last_undegraded":"2024-03-16T19:56:44.880005+0000","last_fullsized":"2024-03-16T19:56:44.880005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884174+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T19:56:44.884174+0000","last_peered":"2024-03-16T19:56:44.884174+0000","last_clean":"2024-03-16T19:56:44.884174+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T19:56:44.884174+0000","last_undegraded":"2024-03-16T19:56:44.884174+0000","last_fullsized":"2024-03-16T19:56:44.884174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856795+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T19:56:44.856795+0000","last_peered":"2024-03-16T19:56:44.856795+0000","last_clean":"2024-03-16T19:56:44.856795+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T19:56:44.856795+0000","last_undegraded":"2024-03-16T19:56:44.856795+0000","last_fullsized":"2024-03-16T19:56:44.856795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880850+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T19:56:44.880850+0000","last_peered":"2024-03-16T19:56:44.880850+0000","last_clean":"2024-03-16T19:56:44.880850+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T19:56:44.880850+0000","last_undegraded":"2024-03-16T19:56:44.880850+0000","last_fullsized":"2024-03-16T19:56:44.880850+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887369+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T19:56:44.887369+0000","last_peered":"2024-03-16T19:56:44.887369+0000","last_clean":"2024-03-16T19:56:44.887369+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T19:56:44.887369+0000","last_undegraded":"2024-03-16T19:56:44.887369+0000","last_fullsized":"2024-03-16T19:56:44.887369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881020+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T19:56:44.881020+0000","last_peered":"2024-03-16T19:56:44.881020+0000","last_clean":"2024-03-16T19:56:44.881020+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T19:56:44.881020+0000","last_undegraded":"2024-03-16T19:56:44.881020+0000","last_fullsized":"2024-03-16T19:56:44.881020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.878982+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T19:56:44.878982+0000","last_peered":"2024-03-16T19:56:44.878982+0000","last_clean":"2024-03-16T19:56:44.878982+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T19:56:44.878982+0000","last_undegraded":"2024-03-16T19:56:44.878982+0000","last_fullsized":"2024-03-16T19:56:44.878982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.886605+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T19:56:44.886605+0000","last_peered":"2024-03-16T19:56:44.886605+0000","last_clean":"2024-03-16T19:56:44.886605+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T19:56:44.886605+0000","last_undegraded":"2024-03-16T19:56:44.886605+0000","last_fullsized":"2024-03-16T19:56:44.886605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869965+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T19:56:44.869965+0000","last_peered":"2024-03-16T19:56:44.869965+0000","last_clean":"2024-03-16T19:56:44.869965+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T19:56:44.869965+0000","last_undegraded":"2024-03-16T19:56:44.869965+0000","last_fullsized":"2024-03-16T19:56:44.869965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868365+0000","last_change":"2024-03-16T19:56:43.466274+0000","last_active":"2024-03-16T19:56:44.868365+0000","last_peered":"2024-03-16T19:56:44.868365+0000","last_clean":"2024-03-16T19:56:44.868365+0000","last_became_active":"2024-03-16T19:56:43.465280+0000","last_became_peered":"2024-03-16T19:56:43.465280+0000","last_unstale":"2024-03-16T19:56:44.868365+0000","last_undegraded":"2024-03-16T19:56:44.868365+0000","last_fullsized":"2024-03-16T19:56:44.868365+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:53.230821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881133+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T19:56:44.881133+0000","last_peered":"2024-03-16T19:56:44.881133+0000","last_clean":"2024-03-16T19:56:44.881133+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T19:56:44.881133+0000","last_undegraded":"2024-03-16T19:56:44.881133+0000","last_fullsized":"2024-03-16T19:56:44.881133+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":38,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.588128+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T19:57:15.588128+0000","last_peered":"2024-03-16T19:57:15.588128+0000","last_clean":"2024-03-16T19:57:15.588128+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T19:57:15.588128+0000","last_undegraded":"2024-03-16T19:57:15.588128+0000","last_fullsized":"2024-03-16T19:57:15.588128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881060+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T19:56:44.881060+0000","last_peered":"2024-03-16T19:56:44.881060+0000","last_clean":"2024-03-16T19:56:44.881060+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T19:56:44.881060+0000","last_undegraded":"2024-03-16T19:56:44.881060+0000","last_fullsized":"2024-03-16T19:56:44.881060+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879400+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T19:56:44.879400+0000","last_peered":"2024-03-16T19:56:44.879400+0000","last_clean":"2024-03-16T19:56:44.879400+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T19:56:44.879400+0000","last_undegraded":"2024-03-16T19:56:44.879400+0000","last_fullsized":"2024-03-16T19:56:44.879400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879992+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T19:56:44.879992+0000","last_peered":"2024-03-16T19:56:44.879992+0000","last_clean":"2024-03-16T19:56:44.879992+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T19:56:44.879992+0000","last_undegraded":"2024-03-16T19:56:44.879992+0000","last_fullsized":"2024-03-16T19:56:44.879992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784460+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T19:56:44.784460+0000","last_peered":"2024-03-16T19:56:44.784460+0000","last_clean":"2024-03-16T19:56:44.784460+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T19:56:44.784460+0000","last_undegraded":"2024-03-16T19:56:44.784460+0000","last_fullsized":"2024-03-16T19:56:44.784460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882782+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T19:56:44.882782+0000","last_peered":"2024-03-16T19:56:44.882782+0000","last_clean":"2024-03-16T19:56:44.882782+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T19:56:44.882782+0000","last_undegraded":"2024-03-16T19:56:44.882782+0000","last_fullsized":"2024-03-16T19:56:44.882782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885731+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T19:56:44.885731+0000","last_peered":"2024-03-16T19:56:44.885731+0000","last_clean":"2024-03-16T19:56:44.885731+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T19:56:44.885731+0000","last_undegraded":"2024-03-16T19:56:44.885731+0000","last_fullsized":"2024-03-16T19:56:44.885731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782625+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T19:56:44.782625+0000","last_peered":"2024-03-16T19:56:44.782625+0000","last_clean":"2024-03-16T19:56:44.782625+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T19:56:44.782625+0000","last_undegraded":"2024-03-16T19:56:44.782625+0000","last_fullsized":"2024-03-16T19:56:44.782625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881525+0000","last_change":"2024-03-16T19:56:36.323272+0000","last_active":"2024-03-16T19:56:44.881525+0000","last_peered":"2024-03-16T19:56:44.881525+0000","last_clean":"2024-03-16T19:56:44.881525+0000","last_became_active":"2024-03-16T19:56:36.322243+0000","last_became_peered":"2024-03-16T19:56:36.322243+0000","last_unstale":"2024-03-16T19:56:44.881525+0000","last_undegraded":"2024-03-16T19:56:44.881525+0000","last_fullsized":"2024-03-16T19:56:44.881525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:49:15.281453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.462053+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T19:57:15.462053+0000","last_peered":"2024-03-16T19:57:15.462053+0000","last_clean":"2024-03-16T19:57:15.462053+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T19:57:15.462053+0000","last_undegraded":"2024-03-16T19:57:15.462053+0000","last_fullsized":"2024-03-16T19:57:15.462053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885201+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T19:56:44.885201+0000","last_peered":"2024-03-16T19:56:44.885201+0000","last_clean":"2024-03-16T19:56:44.885201+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T19:56:44.885201+0000","last_undegraded":"2024-03-16T19:56:44.885201+0000","last_fullsized":"2024-03-16T19:56:44.885201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884450+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T19:56:44.884450+0000","last_peered":"2024-03-16T19:56:44.884450+0000","last_clean":"2024-03-16T19:56:44.884450+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T19:56:44.884450+0000","last_undegraded":"2024-03-16T19:56:44.884450+0000","last_fullsized":"2024-03-16T19:56:44.884450+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'10","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:54.673319+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T19:56:54.673319+0000","last_peered":"2024-03-16T19:56:54.673319+0000","last_clean":"2024-03-16T19:56:54.673319+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T19:56:54.673319+0000","last_undegraded":"2024-03-16T19:56:54.673319+0000","last_fullsized":"2024-03-16T19:56:54.673319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856372+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T19:56:44.856372+0000","last_peered":"2024-03-16T19:56:44.856372+0000","last_clean":"2024-03-16T19:56:44.856372+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T19:56:44.856372+0000","last_undegraded":"2024-03-16T19:56:44.856372+0000","last_fullsized":"2024-03-16T19:56:44.856372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871108+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T19:56:44.871108+0000","last_peered":"2024-03-16T19:56:44.871108+0000","last_clean":"2024-03-16T19:56:44.871108+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T19:56:44.871108+0000","last_undegraded":"2024-03-16T19:56:44.871108+0000","last_fullsized":"2024-03-16T19:56:44.871108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856638+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T19:56:44.856638+0000","last_peered":"2024-03-16T19:56:44.856638+0000","last_clean":"2024-03-16T19:56:44.856638+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T19:56:44.856638+0000","last_undegraded":"2024-03-16T19:56:44.856638+0000","last_fullsized":"2024-03-16T19:56:44.856638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869123+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T19:56:44.869123+0000","last_peered":"2024-03-16T19:56:44.869123+0000","last_clean":"2024-03-16T19:56:44.869123+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T19:56:44.869123+0000","last_undegraded":"2024-03-16T19:56:44.869123+0000","last_fullsized":"2024-03-16T19:56:44.869123+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880901+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T19:56:44.880901+0000","last_peered":"2024-03-16T19:56:44.880901+0000","last_clean":"2024-03-16T19:56:44.880901+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T19:56:44.880901+0000","last_undegraded":"2024-03-16T19:56:44.880901+0000","last_fullsized":"2024-03-16T19:56:44.880901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884122+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T19:56:44.884122+0000","last_peered":"2024-03-16T19:56:44.884122+0000","last_clean":"2024-03-16T19:56:44.884122+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T19:56:44.884122+0000","last_undegraded":"2024-03-16T19:56:44.884122+0000","last_fullsized":"2024-03-16T19:56:44.884122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783021+0000","last_change":"2024-03-16T19:56:43.460381+0000","last_active":"2024-03-16T19:56:44.783021+0000","last_peered":"2024-03-16T19:56:44.783021+0000","last_clean":"2024-03-16T19:56:44.783021+0000","last_became_active":"2024-03-16T19:56:43.459928+0000","last_became_peered":"2024-03-16T19:56:43.459928+0000","last_unstale":"2024-03-16T19:56:44.783021+0000","last_undegraded":"2024-03-16T19:56:44.783021+0000","last_fullsized":"2024-03-16T19:56:44.783021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:06:01.829651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883713+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T19:56:44.883713+0000","last_peered":"2024-03-16T19:56:44.883713+0000","last_clean":"2024-03-16T19:56:44.883713+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T19:56:44.883713+0000","last_undegraded":"2024-03-16T19:56:44.883713+0000","last_fullsized":"2024-03-16T19:56:44.883713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881536+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T19:56:44.881536+0000","last_peered":"2024-03-16T19:56:44.881536+0000","last_clean":"2024-03-16T19:56:44.881536+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T19:56:44.881536+0000","last_undegraded":"2024-03-16T19:56:44.881536+0000","last_fullsized":"2024-03-16T19:56:44.881536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782932+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T19:56:44.782932+0000","last_peered":"2024-03-16T19:56:44.782932+0000","last_clean":"2024-03-16T19:56:44.782932+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T19:56:44.782932+0000","last_undegraded":"2024-03-16T19:56:44.782932+0000","last_fullsized":"2024-03-16T19:56:44.782932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880633+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T19:56:44.880633+0000","last_peered":"2024-03-16T19:56:44.880633+0000","last_clean":"2024-03-16T19:56:44.880633+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T19:56:44.880633+0000","last_undegraded":"2024-03-16T19:56:44.880633+0000","last_fullsized":"2024-03-16T19:56:44.880633+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855564+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T19:56:44.855564+0000","last_peered":"2024-03-16T19:56:44.855564+0000","last_clean":"2024-03-16T19:56:44.855564+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T19:56:44.855564+0000","last_undegraded":"2024-03-16T19:56:44.855564+0000","last_fullsized":"2024-03-16T19:56:44.855564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880999+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T19:56:44.880999+0000","last_peered":"2024-03-16T19:56:44.880999+0000","last_clean":"2024-03-16T19:56:44.880999+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T19:56:44.880999+0000","last_undegraded":"2024-03-16T19:56:44.880999+0000","last_fullsized":"2024-03-16T19:56:44.880999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881302+0000","last_change":"2024-03-16T19:56:43.461007+0000","last_active":"2024-03-16T19:56:44.881302+0000","last_peered":"2024-03-16T19:56:44.881302+0000","last_clean":"2024-03-16T19:56:44.881302+0000","last_became_active":"2024-03-16T19:56:43.460288+0000","last_became_peered":"2024-03-16T19:56:43.460288+0000","last_unstale":"2024-03-16T19:56:44.881302+0000","last_undegraded":"2024-03-16T19:56:44.881302+0000","last_fullsized":"2024-03-16T19:56:44.881302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:58.005225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783384+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T19:56:44.783384+0000","last_peered":"2024-03-16T19:56:44.783384+0000","last_clean":"2024-03-16T19:56:44.783384+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T19:56:44.783384+0000","last_undegraded":"2024-03-16T19:56:44.783384+0000","last_fullsized":"2024-03-16T19:56:44.783384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.385154+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T19:57:18.385154+0000","last_peered":"2024-03-16T19:57:18.385154+0000","last_clean":"2024-03-16T19:57:18.385154+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T19:57:18.385154+0000","last_undegraded":"2024-03-16T19:57:18.385154+0000","last_fullsized":"2024-03-16T19:57:18.385154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880798+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T19:56:44.880798+0000","last_peered":"2024-03-16T19:56:44.880798+0000","last_clean":"2024-03-16T19:56:44.880798+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T19:56:44.880798+0000","last_undegraded":"2024-03-16T19:56:44.880798+0000","last_fullsized":"2024-03-16T19:56:44.880798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881333+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T19:56:44.881333+0000","last_peered":"2024-03-16T19:56:44.881333+0000","last_clean":"2024-03-16T19:56:44.881333+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T19:56:44.881333+0000","last_undegraded":"2024-03-16T19:56:44.881333+0000","last_fullsized":"2024-03-16T19:56:44.881333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880174+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T19:56:44.880174+0000","last_peered":"2024-03-16T19:56:44.880174+0000","last_clean":"2024-03-16T19:56:44.880174+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T19:56:44.880174+0000","last_undegraded":"2024-03-16T19:56:44.880174+0000","last_fullsized":"2024-03-16T19:56:44.880174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.585324+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T19:57:15.585324+0000","last_peered":"2024-03-16T19:57:15.585324+0000","last_clean":"2024-03-16T19:57:15.585324+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T19:57:15.585324+0000","last_undegraded":"2024-03-16T19:57:15.585324+0000","last_fullsized":"2024-03-16T19:57:15.585324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869247+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T19:56:44.869247+0000","last_peered":"2024-03-16T19:56:44.869247+0000","last_clean":"2024-03-16T19:56:44.869247+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T19:56:44.869247+0000","last_undegraded":"2024-03-16T19:56:44.869247+0000","last_fullsized":"2024-03-16T19:56:44.869247+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785098+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T19:56:44.785098+0000","last_peered":"2024-03-16T19:56:44.785098+0000","last_clean":"2024-03-16T19:56:44.785098+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T19:56:44.785098+0000","last_undegraded":"2024-03-16T19:56:44.785098+0000","last_fullsized":"2024-03-16T19:56:44.785098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879070+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T19:56:44.879070+0000","last_peered":"2024-03-16T19:56:44.879070+0000","last_clean":"2024-03-16T19:56:44.879070+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T19:56:44.879070+0000","last_undegraded":"2024-03-16T19:56:44.879070+0000","last_fullsized":"2024-03-16T19:56:44.879070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879660+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T19:56:44.879660+0000","last_peered":"2024-03-16T19:56:44.879660+0000","last_clean":"2024-03-16T19:56:44.879660+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T19:56:44.879660+0000","last_undegraded":"2024-03-16T19:56:44.879660+0000","last_fullsized":"2024-03-16T19:56:44.879660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883871+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T19:56:44.883871+0000","last_peered":"2024-03-16T19:56:44.883871+0000","last_clean":"2024-03-16T19:56:44.883871+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T19:56:44.883871+0000","last_undegraded":"2024-03-16T19:56:44.883871+0000","last_fullsized":"2024-03-16T19:56:44.883871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881090+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T19:56:44.881090+0000","last_peered":"2024-03-16T19:56:44.881090+0000","last_clean":"2024-03-16T19:56:44.881090+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T19:56:44.881090+0000","last_undegraded":"2024-03-16T19:56:44.881090+0000","last_fullsized":"2024-03-16T19:56:44.881090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784903+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T19:56:44.784903+0000","last_peered":"2024-03-16T19:56:44.784903+0000","last_clean":"2024-03-16T19:56:44.784903+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T19:56:44.784903+0000","last_undegraded":"2024-03-16T19:56:44.784903+0000","last_fullsized":"2024-03-16T19:56:44.784903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856454+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T19:56:44.856454+0000","last_peered":"2024-03-16T19:56:44.856454+0000","last_clean":"2024-03-16T19:56:44.856454+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T19:56:44.856454+0000","last_undegraded":"2024-03-16T19:56:44.856454+0000","last_fullsized":"2024-03-16T19:56:44.856454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883722+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T19:56:44.883722+0000","last_peered":"2024-03-16T19:56:44.883722+0000","last_clean":"2024-03-16T19:56:44.883722+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T19:56:44.883722+0000","last_undegraded":"2024-03-16T19:56:44.883722+0000","last_fullsized":"2024-03-16T19:56:44.883722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784862+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T19:56:44.784862+0000","last_peered":"2024-03-16T19:56:44.784862+0000","last_clean":"2024-03-16T19:56:44.784862+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T19:56:44.784862+0000","last_undegraded":"2024-03-16T19:56:44.784862+0000","last_fullsized":"2024-03-16T19:56:44.784862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880645+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T19:56:44.880645+0000","last_peered":"2024-03-16T19:56:44.880645+0000","last_clean":"2024-03-16T19:56:44.880645+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T19:56:44.880645+0000","last_undegraded":"2024-03-16T19:56:44.880645+0000","last_fullsized":"2024-03-16T19:56:44.880645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783243+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T19:56:44.783243+0000","last_peered":"2024-03-16T19:56:44.783243+0000","last_clean":"2024-03-16T19:56:44.783243+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T19:56:44.783243+0000","last_undegraded":"2024-03-16T19:56:44.783243+0000","last_fullsized":"2024-03-16T19:56:44.783243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883489+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T19:56:44.883489+0000","last_peered":"2024-03-16T19:56:44.883489+0000","last_clean":"2024-03-16T19:56:44.883489+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T19:56:44.883489+0000","last_undegraded":"2024-03-16T19:56:44.883489+0000","last_fullsized":"2024-03-16T19:56:44.883489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880427+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T19:56:44.880427+0000","last_peered":"2024-03-16T19:56:44.880427+0000","last_clean":"2024-03-16T19:56:44.880427+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T19:56:44.880427+0000","last_undegraded":"2024-03-16T19:56:44.880427+0000","last_fullsized":"2024-03-16T19:56:44.880427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881350+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T19:56:44.881350+0000","last_peered":"2024-03-16T19:56:44.881350+0000","last_clean":"2024-03-16T19:56:44.881350+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T19:56:44.881350+0000","last_undegraded":"2024-03-16T19:56:44.881350+0000","last_fullsized":"2024-03-16T19:56:44.881350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885426+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T19:56:44.885426+0000","last_peered":"2024-03-16T19:56:44.885426+0000","last_clean":"2024-03-16T19:56:44.885426+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T19:56:44.885426+0000","last_undegraded":"2024-03-16T19:56:44.885426+0000","last_fullsized":"2024-03-16T19:56:44.885426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880885+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T19:56:44.880885+0000","last_peered":"2024-03-16T19:56:44.880885+0000","last_clean":"2024-03-16T19:56:44.880885+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T19:56:44.880885+0000","last_undegraded":"2024-03-16T19:56:44.880885+0000","last_fullsized":"2024-03-16T19:56:44.880885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885871+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T19:56:44.885871+0000","last_peered":"2024-03-16T19:56:44.885871+0000","last_clean":"2024-03-16T19:56:44.885871+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T19:56:44.885871+0000","last_undegraded":"2024-03-16T19:56:44.885871+0000","last_fullsized":"2024-03-16T19:56:44.885871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880278+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T19:56:44.880278+0000","last_peered":"2024-03-16T19:56:44.880278+0000","last_clean":"2024-03-16T19:56:44.880278+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T19:56:44.880278+0000","last_undegraded":"2024-03-16T19:56:44.880278+0000","last_fullsized":"2024-03-16T19:56:44.880278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855681+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T19:56:44.855681+0000","last_peered":"2024-03-16T19:56:44.855681+0000","last_clean":"2024-03-16T19:56:44.855681+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T19:56:44.855681+0000","last_undegraded":"2024-03-16T19:56:44.855681+0000","last_fullsized":"2024-03-16T19:56:44.855681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881487+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T19:56:44.881487+0000","last_peered":"2024-03-16T19:56:44.881487+0000","last_clean":"2024-03-16T19:56:44.881487+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T19:56:44.881487+0000","last_undegraded":"2024-03-16T19:56:44.881487+0000","last_fullsized":"2024-03-16T19:56:44.881487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783246+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T19:56:44.783246+0000","last_peered":"2024-03-16T19:56:44.783246+0000","last_clean":"2024-03-16T19:56:44.783246+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T19:56:44.783246+0000","last_undegraded":"2024-03-16T19:56:44.783246+0000","last_fullsized":"2024-03-16T19:56:44.783246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871059+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T19:56:44.871059+0000","last_peered":"2024-03-16T19:56:44.871059+0000","last_clean":"2024-03-16T19:56:44.871059+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T19:56:44.871059+0000","last_undegraded":"2024-03-16T19:56:44.871059+0000","last_fullsized":"2024-03-16T19:56:44.871059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882578+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T19:56:44.882578+0000","last_peered":"2024-03-16T19:56:44.882578+0000","last_clean":"2024-03-16T19:56:44.882578+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T19:56:44.882578+0000","last_undegraded":"2024-03-16T19:56:44.882578+0000","last_fullsized":"2024-03-16T19:56:44.882578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883483+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T19:56:44.883483+0000","last_peered":"2024-03-16T19:56:44.883483+0000","last_clean":"2024-03-16T19:56:44.883483+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T19:56:44.883483+0000","last_undegraded":"2024-03-16T19:56:44.883483+0000","last_fullsized":"2024-03-16T19:56:44.883483+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887376+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T19:56:44.887376+0000","last_peered":"2024-03-16T19:56:44.887376+0000","last_clean":"2024-03-16T19:56:44.887376+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T19:56:44.887376+0000","last_undegraded":"2024-03-16T19:56:44.887376+0000","last_fullsized":"2024-03-16T19:56:44.887376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784039+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T19:56:44.784039+0000","last_peered":"2024-03-16T19:56:44.784039+0000","last_clean":"2024-03-16T19:56:44.784039+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T19:56:44.784039+0000","last_undegraded":"2024-03-16T19:56:44.784039+0000","last_fullsized":"2024-03-16T19:56:44.784039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880860+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T19:56:44.880860+0000","last_peered":"2024-03-16T19:56:44.880860+0000","last_clean":"2024-03-16T19:56:44.880860+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T19:56:44.880860+0000","last_undegraded":"2024-03-16T19:56:44.880860+0000","last_fullsized":"2024-03-16T19:56:44.880860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883261+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T19:56:44.883261+0000","last_peered":"2024-03-16T19:56:44.883261+0000","last_clean":"2024-03-16T19:56:44.883261+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T19:56:44.883261+0000","last_undegraded":"2024-03-16T19:56:44.883261+0000","last_fullsized":"2024-03-16T19:56:44.883261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882289+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T19:56:44.882289+0000","last_peered":"2024-03-16T19:56:44.882289+0000","last_clean":"2024-03-16T19:56:44.882289+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T19:56:44.882289+0000","last_undegraded":"2024-03-16T19:56:44.882289+0000","last_fullsized":"2024-03-16T19:56:44.882289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883418+0000","last_change":"2024-03-16T19:56:36.306936+0000","last_active":"2024-03-16T19:56:44.883418+0000","last_peered":"2024-03-16T19:56:44.883418+0000","last_clean":"2024-03-16T19:56:44.883418+0000","last_became_active":"2024-03-16T19:56:36.306436+0000","last_became_peered":"2024-03-16T19:56:36.306436+0000","last_unstale":"2024-03-16T19:56:44.883418+0000","last_undegraded":"2024-03-16T19:56:44.883418+0000","last_fullsized":"2024-03-16T19:56:44.883418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879751+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T19:56:44.879751+0000","last_peered":"2024-03-16T19:56:44.879751+0000","last_clean":"2024-03-16T19:56:44.879751+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T19:56:44.879751+0000","last_undegraded":"2024-03-16T19:56:44.879751+0000","last_fullsized":"2024-03-16T19:56:44.879751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856790+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T19:56:44.856790+0000","last_peered":"2024-03-16T19:56:44.856790+0000","last_clean":"2024-03-16T19:56:44.856790+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T19:56:44.856790+0000","last_undegraded":"2024-03-16T19:56:44.856790+0000","last_fullsized":"2024-03-16T19:56:44.856790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887093+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T19:56:44.887093+0000","last_peered":"2024-03-16T19:56:44.887093+0000","last_clean":"2024-03-16T19:56:44.887093+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T19:56:44.887093+0000","last_undegraded":"2024-03-16T19:56:44.887093+0000","last_fullsized":"2024-03-16T19:56:44.887093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5262,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":9924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476756,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":901120,"data_stored":714188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476758,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":127080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476757,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":123328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476755,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":123364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":323584,"data_stored":127702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":123238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509461,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":913408,"data_stored":714266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":311296,"data_stored":123398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:57:18.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:57:19.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 Processor -- start 2024-03-16T19:57:19.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 -- start start 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0f1c151000 0x7f0f1c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0f1c0c36b0 0x7f0f1c15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c072f10 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c0c5000 con 0x7f0f1c151000 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f210d6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0c8ec0 con 0x7f0f1c0c8980 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f13fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0f1c0c36b0 0x7f0f1c15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f13fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0f1c0c36b0 0x7f0f1c15a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59436/0 (socket says 172.21.15.32:59436) 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f13fff640 1 -- 172.21.15.32:0/1478270020 learned_addr learned my addr 172.21.15.32:0/1478270020 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f1b59e640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.088+0000 7f0f1ad9d640 1 --2- 172.21.15.32:0/1478270020 >> v2:172.21.15.59:3300/0 conn(0x7f0f1c151000 0x7f0f1c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.092+0000 7f0f13fff640 1 -- 172.21.15.32:0/1478270020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 msgr2=0x7f0f1c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.092+0000 7f0f13fff640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.092+0000 7f0f13fff640 1 -- 172.21.15.32:0/1478270020 >> v2:172.21.15.59:3300/0 conn(0x7f0f1c151000 msgr2=0x7f0f1c0c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.092+0000 7f0f13fff640 1 --2- 172.21.15.32:0/1478270020 >> v2:172.21.15.59:3300/0 conn(0x7f0f1c151000 0x7f0f1c0c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.092+0000 7f0f13fff640 1 -- 172.21.15.32:0/1478270020 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c072950 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.096+0000 7f0f13fff640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c15a240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0725d0 tx=0x7f0f0401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d6b6b627919f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f1a59c640 1 -- 172.21.15.32:0/1478270020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040aa640 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f1a59c640 1 -- 172.21.15.32:0/1478270020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f040aa7a0 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f1a59c640 1 -- 172.21.15.32:0/1478270020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040aab30 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- 172.21.15.32:0/1478270020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c0c36b0 msgr2=0x7f0f1c15a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c15a240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0725d0 tx=0x7f0f0401c520 comp rx=0 tx=0).stop 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- 172.21.15.32:0/1478270020 shutdown_connections 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 --2- 172.21.15.32:0/1478270020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c15a240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 --2- 172.21.15.32:0/1478270020 >> v2:172.21.15.59:3300/0 conn(0x7f0f1c151000 0x7f0f1c0c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- 172.21.15.32:0/1478270020 >> 172.21.15.32:0/1478270020 conn(0x7f0f1c06c0d0 msgr2=0x7f0f1c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- 172.21.15.32:0/1478270020 shutdown_connections 2024-03-16T19:57:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- 172.21.15.32:0/1478270020 wait complete. 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 Processor -- start 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.100+0000 7f0f210d6640 1 -- start start 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c0c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c160d10 con 0x7f0f1c151000 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cd2f0 con 0x7f0f1c0c36b0 2024-03-16T19:57:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f210d6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cc750 con 0x7f0f1c0c8980 2024-03-16T19:57:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f1b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f1b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59452/0 (socket says 172.21.15.32:59452) 2024-03-16T19:57:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f1b59e640 1 -- 172.21.15.32:0/471858598 learned_addr learned my addr 172.21.15.32:0/471858598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f13fff640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.104+0000 7f0f1ad9d640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c0c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 -- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 msgr2=0x7f0f1c0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 -- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1c0c36b0 msgr2=0x7f0f1c0c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c0c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 -- 172.21.15.32:0/471858598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f040094d0 con 0x7f0f1c151000 2024-03-16T19:57:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.108+0000 7f0f1b59e640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f14007e60 tx=0x7f0f1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:19.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.112+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14014620 con 0x7f0f1c151000 2024-03-16T19:57:19.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.112+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400ee90 con 0x7f0f1c151000 2024-03-16T19:57:19.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.112+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c0caff0 con 0x7f0f1c151000 2024-03-16T19:57:19.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.112+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f140149e0 con 0x7f0f1c151000 2024-03-16T19:57:19.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.112+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c0ccca0 con 0x7f0f1c151000 2024-03-16T19:57:19.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.116+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c068000 con 0x7f0f1c151000 2024-03-16T19:57:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.132+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0f14014b40 con 0x7f0f1c151000 2024-03-16T19:57:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.132+0000 7f0f137fe640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 0x7f0efc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.132+0000 7f0f1ad9d640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 0x7f0efc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.136+0000 7f0f1ad9d640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 0x7f0efc069330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0079c0 tx=0x7f0f0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:19.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.136+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0f14056d20 con 0x7f0f1c151000 2024-03-16T19:57:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.424+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0f14052fc0 con 0x7f0f1c151000 2024-03-16T19:57:19.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.592+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f1c15b190 con 0x7f0efc066e70 2024-03-16T19:57:19.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.608+0000 7f0f137fe640 1 -- 172.21.15.32:0/471858598 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332571 (secure 0 0 0) 0x7f0f1c15b190 con 0x7f0efc066e70 2024-03-16T19:57:19.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:19.583 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 msgr2=0x7f0efc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 0x7f0efc069330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0079c0 tx=0x7f0f0c008040 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 msgr2=0x7f0f1c1f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f14007e60 tx=0x7f0f1400e4b0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 shutdown_connections 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0efc066e70 0x7f0efc069330 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f1c151000 0x7f0f1c1f30f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f1c0c8980 0x7f0f1c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 --2- 172.21.15.32:0/471858598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1c0c36b0 0x7f0f1c0c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 >> 172.21.15.32:0/471858598 conn(0x7f0f1c06c0d0 msgr2=0x7f0f1c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:19.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 shutdown_connections 2024-03-16T19:57:19.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.612+0000 7f0f210d6640 1 -- 172.21.15.32:0/471858598 wait complete. 2024-03-16T19:57:19.599 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-16T19:57:18.578660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595630,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":219,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236404,"kb_used_data":3604,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738316,"statfs":{"total":773094113280,"available":772852035584,"internally_reserved":0,"allocated":3690496,"data_stored":2176564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":622,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881323+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T19:56:44.881323+0000","last_peered":"2024-03-16T19:56:44.881323+0000","last_clean":"2024-03-16T19:56:44.881323+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T19:56:44.881323+0000","last_undegraded":"2024-03-16T19:56:44.881323+0000","last_fullsized":"2024-03-16T19:56:44.881323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784985+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T19:56:44.784985+0000","last_peered":"2024-03-16T19:56:44.784985+0000","last_clean":"2024-03-16T19:56:44.784985+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T19:56:44.784985+0000","last_undegraded":"2024-03-16T19:56:44.784985+0000","last_fullsized":"2024-03-16T19:56:44.784985+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782560+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T19:56:44.782560+0000","last_peered":"2024-03-16T19:56:44.782560+0000","last_clean":"2024-03-16T19:56:44.782560+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T19:56:44.782560+0000","last_undegraded":"2024-03-16T19:56:44.782560+0000","last_fullsized":"2024-03-16T19:56:44.782560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785841+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T19:56:44.785841+0000","last_peered":"2024-03-16T19:56:44.785841+0000","last_clean":"2024-03-16T19:56:44.785841+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T19:56:44.785841+0000","last_undegraded":"2024-03-16T19:56:44.785841+0000","last_fullsized":"2024-03-16T19:56:44.785841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880422+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T19:56:44.880422+0000","last_peered":"2024-03-16T19:56:44.880422+0000","last_clean":"2024-03-16T19:56:44.880422+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T19:56:44.880422+0000","last_undegraded":"2024-03-16T19:56:44.880422+0000","last_fullsized":"2024-03-16T19:56:44.880422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868804+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T19:56:44.868804+0000","last_peered":"2024-03-16T19:56:44.868804+0000","last_clean":"2024-03-16T19:56:44.868804+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T19:56:44.868804+0000","last_undegraded":"2024-03-16T19:56:44.868804+0000","last_fullsized":"2024-03-16T19:56:44.868804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885168+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T19:56:44.885168+0000","last_peered":"2024-03-16T19:56:44.885168+0000","last_clean":"2024-03-16T19:56:44.885168+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T19:56:44.885168+0000","last_undegraded":"2024-03-16T19:56:44.885168+0000","last_fullsized":"2024-03-16T19:56:44.885168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883804+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T19:56:44.883804+0000","last_peered":"2024-03-16T19:56:44.883804+0000","last_clean":"2024-03-16T19:56:44.883804+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T19:56:44.883804+0000","last_undegraded":"2024-03-16T19:56:44.883804+0000","last_fullsized":"2024-03-16T19:56:44.883804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869676+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T19:56:44.869676+0000","last_peered":"2024-03-16T19:56:44.869676+0000","last_clean":"2024-03-16T19:56:44.869676+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T19:56:44.869676+0000","last_undegraded":"2024-03-16T19:56:44.869676+0000","last_fullsized":"2024-03-16T19:56:44.869676+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883912+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T19:56:44.883912+0000","last_peered":"2024-03-16T19:56:44.883912+0000","last_clean":"2024-03-16T19:56:44.883912+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T19:56:44.883912+0000","last_undegraded":"2024-03-16T19:56:44.883912+0000","last_fullsized":"2024-03-16T19:56:44.883912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884919+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T19:56:44.884919+0000","last_peered":"2024-03-16T19:56:44.884919+0000","last_clean":"2024-03-16T19:56:44.884919+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T19:56:44.884919+0000","last_undegraded":"2024-03-16T19:56:44.884919+0000","last_fullsized":"2024-03-16T19:56:44.884919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869257+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T19:56:44.869257+0000","last_peered":"2024-03-16T19:56:44.869257+0000","last_clean":"2024-03-16T19:56:44.869257+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T19:56:44.869257+0000","last_undegraded":"2024-03-16T19:56:44.869257+0000","last_fullsized":"2024-03-16T19:56:44.869257+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880296+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T19:56:44.880296+0000","last_peered":"2024-03-16T19:56:44.880296+0000","last_clean":"2024-03-16T19:56:44.880296+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T19:56:44.880296+0000","last_undegraded":"2024-03-16T19:56:44.880296+0000","last_fullsized":"2024-03-16T19:56:44.880296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883063+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T19:56:44.883063+0000","last_peered":"2024-03-16T19:56:44.883063+0000","last_clean":"2024-03-16T19:56:44.883063+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T19:56:44.883063+0000","last_undegraded":"2024-03-16T19:56:44.883063+0000","last_fullsized":"2024-03-16T19:56:44.883063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870662+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T19:56:44.870662+0000","last_peered":"2024-03-16T19:56:44.870662+0000","last_clean":"2024-03-16T19:56:44.870662+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T19:56:44.870662+0000","last_undegraded":"2024-03-16T19:56:44.870662+0000","last_fullsized":"2024-03-16T19:56:44.870662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783555+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T19:56:44.783555+0000","last_peered":"2024-03-16T19:56:44.783555+0000","last_clean":"2024-03-16T19:56:44.783555+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T19:56:44.783555+0000","last_undegraded":"2024-03-16T19:56:44.783555+0000","last_fullsized":"2024-03-16T19:56:44.783555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856445+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T19:56:44.856445+0000","last_peered":"2024-03-16T19:56:44.856445+0000","last_clean":"2024-03-16T19:56:44.856445+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T19:56:44.856445+0000","last_undegraded":"2024-03-16T19:56:44.856445+0000","last_fullsized":"2024-03-16T19:56:44.856445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870742+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T19:56:44.870742+0000","last_peered":"2024-03-16T19:56:44.870742+0000","last_clean":"2024-03-16T19:56:44.870742+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T19:56:44.870742+0000","last_undegraded":"2024-03-16T19:56:44.870742+0000","last_fullsized":"2024-03-16T19:56:44.870742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783732+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T19:56:44.783732+0000","last_peered":"2024-03-16T19:56:44.783732+0000","last_clean":"2024-03-16T19:56:44.783732+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T19:56:44.783732+0000","last_undegraded":"2024-03-16T19:56:44.783732+0000","last_fullsized":"2024-03-16T19:56:44.783732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855768+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T19:56:44.855768+0000","last_peered":"2024-03-16T19:56:44.855768+0000","last_clean":"2024-03-16T19:56:44.855768+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T19:56:44.855768+0000","last_undegraded":"2024-03-16T19:56:44.855768+0000","last_fullsized":"2024-03-16T19:56:44.855768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856952+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T19:56:44.856952+0000","last_peered":"2024-03-16T19:56:44.856952+0000","last_clean":"2024-03-16T19:56:44.856952+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T19:56:44.856952+0000","last_undegraded":"2024-03-16T19:56:44.856952+0000","last_fullsized":"2024-03-16T19:56:44.856952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868777+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T19:56:44.868777+0000","last_peered":"2024-03-16T19:56:44.868777+0000","last_clean":"2024-03-16T19:56:44.868777+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T19:56:44.868777+0000","last_undegraded":"2024-03-16T19:56:44.868777+0000","last_fullsized":"2024-03-16T19:56:44.868777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879259+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T19:56:44.879259+0000","last_peered":"2024-03-16T19:56:44.879259+0000","last_clean":"2024-03-16T19:56:44.879259+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T19:56:44.879259+0000","last_undegraded":"2024-03-16T19:56:44.879259+0000","last_fullsized":"2024-03-16T19:56:44.879259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870617+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T19:56:44.870617+0000","last_peered":"2024-03-16T19:56:44.870617+0000","last_clean":"2024-03-16T19:56:44.870617+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T19:56:44.870617+0000","last_undegraded":"2024-03-16T19:56:44.870617+0000","last_fullsized":"2024-03-16T19:56:44.870617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881303+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T19:56:44.881303+0000","last_peered":"2024-03-16T19:56:44.881303+0000","last_clean":"2024-03-16T19:56:44.881303+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T19:56:44.881303+0000","last_undegraded":"2024-03-16T19:56:44.881303+0000","last_fullsized":"2024-03-16T19:56:44.881303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881781+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T19:56:44.881781+0000","last_peered":"2024-03-16T19:56:44.881781+0000","last_clean":"2024-03-16T19:56:44.881781+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T19:56:44.881781+0000","last_undegraded":"2024-03-16T19:56:44.881781+0000","last_fullsized":"2024-03-16T19:56:44.881781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868298+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T19:56:44.868298+0000","last_peered":"2024-03-16T19:56:44.868298+0000","last_clean":"2024-03-16T19:56:44.868298+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T19:56:44.868298+0000","last_undegraded":"2024-03-16T19:56:44.868298+0000","last_fullsized":"2024-03-16T19:56:44.868298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785469+0000","last_change":"2024-03-16T19:56:36.217377+0000","last_active":"2024-03-16T19:56:44.785469+0000","last_peered":"2024-03-16T19:56:44.785469+0000","last_clean":"2024-03-16T19:56:44.785469+0000","last_became_active":"2024-03-16T19:56:36.216579+0000","last_became_peered":"2024-03-16T19:56:36.216579+0000","last_unstale":"2024-03-16T19:56:44.785469+0000","last_undegraded":"2024-03-16T19:56:44.785469+0000","last_fullsized":"2024-03-16T19:56:44.785469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856847+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T19:56:44.856847+0000","last_peered":"2024-03-16T19:56:44.856847+0000","last_clean":"2024-03-16T19:56:44.856847+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T19:56:44.856847+0000","last_undegraded":"2024-03-16T19:56:44.856847+0000","last_fullsized":"2024-03-16T19:56:44.856847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782416+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T19:56:44.782416+0000","last_peered":"2024-03-16T19:56:44.782416+0000","last_clean":"2024-03-16T19:56:44.782416+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T19:56:44.782416+0000","last_undegraded":"2024-03-16T19:56:44.782416+0000","last_fullsized":"2024-03-16T19:56:44.782416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856282+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T19:56:44.856282+0000","last_peered":"2024-03-16T19:56:44.856282+0000","last_clean":"2024-03-16T19:56:44.856282+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T19:56:44.856282+0000","last_undegraded":"2024-03-16T19:56:44.856282+0000","last_fullsized":"2024-03-16T19:56:44.856282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885652+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T19:56:44.885652+0000","last_peered":"2024-03-16T19:56:44.885652+0000","last_clean":"2024-03-16T19:56:44.885652+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T19:56:44.885652+0000","last_undegraded":"2024-03-16T19:56:44.885652+0000","last_fullsized":"2024-03-16T19:56:44.885652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880177+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T19:56:44.880177+0000","last_peered":"2024-03-16T19:56:44.880177+0000","last_clean":"2024-03-16T19:56:44.880177+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T19:56:44.880177+0000","last_undegraded":"2024-03-16T19:56:44.880177+0000","last_fullsized":"2024-03-16T19:56:44.880177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885381+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T19:56:44.885381+0000","last_peered":"2024-03-16T19:56:44.885381+0000","last_clean":"2024-03-16T19:56:44.885381+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T19:56:44.885381+0000","last_undegraded":"2024-03-16T19:56:44.885381+0000","last_fullsized":"2024-03-16T19:56:44.885381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856509+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T19:56:44.856509+0000","last_peered":"2024-03-16T19:56:44.856509+0000","last_clean":"2024-03-16T19:56:44.856509+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T19:56:44.856509+0000","last_undegraded":"2024-03-16T19:56:44.856509+0000","last_fullsized":"2024-03-16T19:56:44.856509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855979+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T19:56:44.855979+0000","last_peered":"2024-03-16T19:56:44.855979+0000","last_clean":"2024-03-16T19:56:44.855979+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T19:56:44.855979+0000","last_undegraded":"2024-03-16T19:56:44.855979+0000","last_fullsized":"2024-03-16T19:56:44.855979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869818+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T19:56:44.869818+0000","last_peered":"2024-03-16T19:56:44.869818+0000","last_clean":"2024-03-16T19:56:44.869818+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T19:56:44.869818+0000","last_undegraded":"2024-03-16T19:56:44.869818+0000","last_fullsized":"2024-03-16T19:56:44.869818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885665+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T19:56:44.885665+0000","last_peered":"2024-03-16T19:56:44.885665+0000","last_clean":"2024-03-16T19:56:44.885665+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T19:56:44.885665+0000","last_undegraded":"2024-03-16T19:56:44.885665+0000","last_fullsized":"2024-03-16T19:56:44.885665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857231+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T19:56:44.857231+0000","last_peered":"2024-03-16T19:56:44.857231+0000","last_clean":"2024-03-16T19:56:44.857231+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T19:56:44.857231+0000","last_undegraded":"2024-03-16T19:56:44.857231+0000","last_fullsized":"2024-03-16T19:56:44.857231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855744+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T19:56:44.855744+0000","last_peered":"2024-03-16T19:56:44.855744+0000","last_clean":"2024-03-16T19:56:44.855744+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T19:56:44.855744+0000","last_undegraded":"2024-03-16T19:56:44.855744+0000","last_fullsized":"2024-03-16T19:56:44.855744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879840+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T19:56:44.879840+0000","last_peered":"2024-03-16T19:56:44.879840+0000","last_clean":"2024-03-16T19:56:44.879840+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T19:56:44.879840+0000","last_undegraded":"2024-03-16T19:56:44.879840+0000","last_fullsized":"2024-03-16T19:56:44.879840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857201+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T19:56:44.857201+0000","last_peered":"2024-03-16T19:56:44.857201+0000","last_clean":"2024-03-16T19:56:44.857201+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T19:56:44.857201+0000","last_undegraded":"2024-03-16T19:56:44.857201+0000","last_fullsized":"2024-03-16T19:56:44.857201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880083+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T19:56:44.880083+0000","last_peered":"2024-03-16T19:56:44.880083+0000","last_clean":"2024-03-16T19:56:44.880083+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T19:56:44.880083+0000","last_undegraded":"2024-03-16T19:56:44.880083+0000","last_fullsized":"2024-03-16T19:56:44.880083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879559+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T19:56:44.879559+0000","last_peered":"2024-03-16T19:56:44.879559+0000","last_clean":"2024-03-16T19:56:44.879559+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T19:56:44.879559+0000","last_undegraded":"2024-03-16T19:56:44.879559+0000","last_fullsized":"2024-03-16T19:56:44.879559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881219+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T19:56:44.881219+0000","last_peered":"2024-03-16T19:56:44.881219+0000","last_clean":"2024-03-16T19:56:44.881219+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T19:56:44.881219+0000","last_undegraded":"2024-03-16T19:56:44.881219+0000","last_fullsized":"2024-03-16T19:56:44.881219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783051+0000","last_change":"2024-03-16T19:56:36.305013+0000","last_active":"2024-03-16T19:56:44.783051+0000","last_peered":"2024-03-16T19:56:44.783051+0000","last_clean":"2024-03-16T19:56:44.783051+0000","last_became_active":"2024-03-16T19:56:36.303533+0000","last_became_peered":"2024-03-16T19:56:36.303533+0000","last_unstale":"2024-03-16T19:56:44.783051+0000","last_undegraded":"2024-03-16T19:56:44.783051+0000","last_fullsized":"2024-03-16T19:56:44.783051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883237+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T19:56:44.883237+0000","last_peered":"2024-03-16T19:56:44.883237+0000","last_clean":"2024-03-16T19:56:44.883237+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T19:56:44.883237+0000","last_undegraded":"2024-03-16T19:56:44.883237+0000","last_fullsized":"2024-03-16T19:56:44.883237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879868+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T19:56:44.879868+0000","last_peered":"2024-03-16T19:56:44.879868+0000","last_clean":"2024-03-16T19:56:44.879868+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T19:56:44.879868+0000","last_undegraded":"2024-03-16T19:56:44.879868+0000","last_fullsized":"2024-03-16T19:56:44.879868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882315+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T19:56:44.882315+0000","last_peered":"2024-03-16T19:56:44.882315+0000","last_clean":"2024-03-16T19:56:44.882315+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T19:56:44.882315+0000","last_undegraded":"2024-03-16T19:56:44.882315+0000","last_fullsized":"2024-03-16T19:56:44.882315+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884701+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T19:56:44.884701+0000","last_peered":"2024-03-16T19:56:44.884701+0000","last_clean":"2024-03-16T19:56:44.884701+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T19:56:44.884701+0000","last_undegraded":"2024-03-16T19:56:44.884701+0000","last_fullsized":"2024-03-16T19:56:44.884701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881140+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T19:56:44.881140+0000","last_peered":"2024-03-16T19:56:44.881140+0000","last_clean":"2024-03-16T19:56:44.881140+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T19:56:44.881140+0000","last_undegraded":"2024-03-16T19:56:44.881140+0000","last_fullsized":"2024-03-16T19:56:44.881140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880005+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T19:56:44.880005+0000","last_peered":"2024-03-16T19:56:44.880005+0000","last_clean":"2024-03-16T19:56:44.880005+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T19:56:44.880005+0000","last_undegraded":"2024-03-16T19:56:44.880005+0000","last_fullsized":"2024-03-16T19:56:44.880005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884174+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T19:56:44.884174+0000","last_peered":"2024-03-16T19:56:44.884174+0000","last_clean":"2024-03-16T19:56:44.884174+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T19:56:44.884174+0000","last_undegraded":"2024-03-16T19:56:44.884174+0000","last_fullsized":"2024-03-16T19:56:44.884174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856795+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T19:56:44.856795+0000","last_peered":"2024-03-16T19:56:44.856795+0000","last_clean":"2024-03-16T19:56:44.856795+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T19:56:44.856795+0000","last_undegraded":"2024-03-16T19:56:44.856795+0000","last_fullsized":"2024-03-16T19:56:44.856795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880850+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T19:56:44.880850+0000","last_peered":"2024-03-16T19:56:44.880850+0000","last_clean":"2024-03-16T19:56:44.880850+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T19:56:44.880850+0000","last_undegraded":"2024-03-16T19:56:44.880850+0000","last_fullsized":"2024-03-16T19:56:44.880850+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887369+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T19:56:44.887369+0000","last_peered":"2024-03-16T19:56:44.887369+0000","last_clean":"2024-03-16T19:56:44.887369+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T19:56:44.887369+0000","last_undegraded":"2024-03-16T19:56:44.887369+0000","last_fullsized":"2024-03-16T19:56:44.887369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881020+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T19:56:44.881020+0000","last_peered":"2024-03-16T19:56:44.881020+0000","last_clean":"2024-03-16T19:56:44.881020+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T19:56:44.881020+0000","last_undegraded":"2024-03-16T19:56:44.881020+0000","last_fullsized":"2024-03-16T19:56:44.881020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.878982+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T19:56:44.878982+0000","last_peered":"2024-03-16T19:56:44.878982+0000","last_clean":"2024-03-16T19:56:44.878982+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T19:56:44.878982+0000","last_undegraded":"2024-03-16T19:56:44.878982+0000","last_fullsized":"2024-03-16T19:56:44.878982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.886605+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T19:56:44.886605+0000","last_peered":"2024-03-16T19:56:44.886605+0000","last_clean":"2024-03-16T19:56:44.886605+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T19:56:44.886605+0000","last_undegraded":"2024-03-16T19:56:44.886605+0000","last_fullsized":"2024-03-16T19:56:44.886605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869965+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T19:56:44.869965+0000","last_peered":"2024-03-16T19:56:44.869965+0000","last_clean":"2024-03-16T19:56:44.869965+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T19:56:44.869965+0000","last_undegraded":"2024-03-16T19:56:44.869965+0000","last_fullsized":"2024-03-16T19:56:44.869965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868365+0000","last_change":"2024-03-16T19:56:43.466274+0000","last_active":"2024-03-16T19:56:44.868365+0000","last_peered":"2024-03-16T19:56:44.868365+0000","last_clean":"2024-03-16T19:56:44.868365+0000","last_became_active":"2024-03-16T19:56:43.465280+0000","last_became_peered":"2024-03-16T19:56:43.465280+0000","last_unstale":"2024-03-16T19:56:44.868365+0000","last_undegraded":"2024-03-16T19:56:44.868365+0000","last_fullsized":"2024-03-16T19:56:44.868365+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:53.230821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881133+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T19:56:44.881133+0000","last_peered":"2024-03-16T19:56:44.881133+0000","last_clean":"2024-03-16T19:56:44.881133+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T19:56:44.881133+0000","last_undegraded":"2024-03-16T19:56:44.881133+0000","last_fullsized":"2024-03-16T19:56:44.881133+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":38,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.588128+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T19:57:15.588128+0000","last_peered":"2024-03-16T19:57:15.588128+0000","last_clean":"2024-03-16T19:57:15.588128+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T19:57:15.588128+0000","last_undegraded":"2024-03-16T19:57:15.588128+0000","last_fullsized":"2024-03-16T19:57:15.588128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881060+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T19:56:44.881060+0000","last_peered":"2024-03-16T19:56:44.881060+0000","last_clean":"2024-03-16T19:56:44.881060+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T19:56:44.881060+0000","last_undegraded":"2024-03-16T19:56:44.881060+0000","last_fullsized":"2024-03-16T19:56:44.881060+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879400+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T19:56:44.879400+0000","last_peered":"2024-03-16T19:56:44.879400+0000","last_clean":"2024-03-16T19:56:44.879400+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T19:56:44.879400+0000","last_undegraded":"2024-03-16T19:56:44.879400+0000","last_fullsized":"2024-03-16T19:56:44.879400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879992+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T19:56:44.879992+0000","last_peered":"2024-03-16T19:56:44.879992+0000","last_clean":"2024-03-16T19:56:44.879992+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T19:56:44.879992+0000","last_undegraded":"2024-03-16T19:56:44.879992+0000","last_fullsized":"2024-03-16T19:56:44.879992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784460+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T19:56:44.784460+0000","last_peered":"2024-03-16T19:56:44.784460+0000","last_clean":"2024-03-16T19:56:44.784460+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T19:56:44.784460+0000","last_undegraded":"2024-03-16T19:56:44.784460+0000","last_fullsized":"2024-03-16T19:56:44.784460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882782+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T19:56:44.882782+0000","last_peered":"2024-03-16T19:56:44.882782+0000","last_clean":"2024-03-16T19:56:44.882782+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T19:56:44.882782+0000","last_undegraded":"2024-03-16T19:56:44.882782+0000","last_fullsized":"2024-03-16T19:56:44.882782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885731+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T19:56:44.885731+0000","last_peered":"2024-03-16T19:56:44.885731+0000","last_clean":"2024-03-16T19:56:44.885731+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T19:56:44.885731+0000","last_undegraded":"2024-03-16T19:56:44.885731+0000","last_fullsized":"2024-03-16T19:56:44.885731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782625+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T19:56:44.782625+0000","last_peered":"2024-03-16T19:56:44.782625+0000","last_clean":"2024-03-16T19:56:44.782625+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T19:56:44.782625+0000","last_undegraded":"2024-03-16T19:56:44.782625+0000","last_fullsized":"2024-03-16T19:56:44.782625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881525+0000","last_change":"2024-03-16T19:56:36.323272+0000","last_active":"2024-03-16T19:56:44.881525+0000","last_peered":"2024-03-16T19:56:44.881525+0000","last_clean":"2024-03-16T19:56:44.881525+0000","last_became_active":"2024-03-16T19:56:36.322243+0000","last_became_peered":"2024-03-16T19:56:36.322243+0000","last_unstale":"2024-03-16T19:56:44.881525+0000","last_undegraded":"2024-03-16T19:56:44.881525+0000","last_fullsized":"2024-03-16T19:56:44.881525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:49:15.281453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.462053+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T19:57:15.462053+0000","last_peered":"2024-03-16T19:57:15.462053+0000","last_clean":"2024-03-16T19:57:15.462053+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T19:57:15.462053+0000","last_undegraded":"2024-03-16T19:57:15.462053+0000","last_fullsized":"2024-03-16T19:57:15.462053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885201+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T19:56:44.885201+0000","last_peered":"2024-03-16T19:56:44.885201+0000","last_clean":"2024-03-16T19:56:44.885201+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T19:56:44.885201+0000","last_undegraded":"2024-03-16T19:56:44.885201+0000","last_fullsized":"2024-03-16T19:56:44.885201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884450+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T19:56:44.884450+0000","last_peered":"2024-03-16T19:56:44.884450+0000","last_clean":"2024-03-16T19:56:44.884450+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T19:56:44.884450+0000","last_undegraded":"2024-03-16T19:56:44.884450+0000","last_fullsized":"2024-03-16T19:56:44.884450+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'10","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:54.673319+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T19:56:54.673319+0000","last_peered":"2024-03-16T19:56:54.673319+0000","last_clean":"2024-03-16T19:56:54.673319+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T19:56:54.673319+0000","last_undegraded":"2024-03-16T19:56:54.673319+0000","last_fullsized":"2024-03-16T19:56:54.673319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856372+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T19:56:44.856372+0000","last_peered":"2024-03-16T19:56:44.856372+0000","last_clean":"2024-03-16T19:56:44.856372+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T19:56:44.856372+0000","last_undegraded":"2024-03-16T19:56:44.856372+0000","last_fullsized":"2024-03-16T19:56:44.856372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871108+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T19:56:44.871108+0000","last_peered":"2024-03-16T19:56:44.871108+0000","last_clean":"2024-03-16T19:56:44.871108+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T19:56:44.871108+0000","last_undegraded":"2024-03-16T19:56:44.871108+0000","last_fullsized":"2024-03-16T19:56:44.871108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856638+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T19:56:44.856638+0000","last_peered":"2024-03-16T19:56:44.856638+0000","last_clean":"2024-03-16T19:56:44.856638+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T19:56:44.856638+0000","last_undegraded":"2024-03-16T19:56:44.856638+0000","last_fullsized":"2024-03-16T19:56:44.856638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869123+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T19:56:44.869123+0000","last_peered":"2024-03-16T19:56:44.869123+0000","last_clean":"2024-03-16T19:56:44.869123+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T19:56:44.869123+0000","last_undegraded":"2024-03-16T19:56:44.869123+0000","last_fullsized":"2024-03-16T19:56:44.869123+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880901+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T19:56:44.880901+0000","last_peered":"2024-03-16T19:56:44.880901+0000","last_clean":"2024-03-16T19:56:44.880901+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T19:56:44.880901+0000","last_undegraded":"2024-03-16T19:56:44.880901+0000","last_fullsized":"2024-03-16T19:56:44.880901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884122+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T19:56:44.884122+0000","last_peered":"2024-03-16T19:56:44.884122+0000","last_clean":"2024-03-16T19:56:44.884122+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T19:56:44.884122+0000","last_undegraded":"2024-03-16T19:56:44.884122+0000","last_fullsized":"2024-03-16T19:56:44.884122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783021+0000","last_change":"2024-03-16T19:56:43.460381+0000","last_active":"2024-03-16T19:56:44.783021+0000","last_peered":"2024-03-16T19:56:44.783021+0000","last_clean":"2024-03-16T19:56:44.783021+0000","last_became_active":"2024-03-16T19:56:43.459928+0000","last_became_peered":"2024-03-16T19:56:43.459928+0000","last_unstale":"2024-03-16T19:56:44.783021+0000","last_undegraded":"2024-03-16T19:56:44.783021+0000","last_fullsized":"2024-03-16T19:56:44.783021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:06:01.829651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883713+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T19:56:44.883713+0000","last_peered":"2024-03-16T19:56:44.883713+0000","last_clean":"2024-03-16T19:56:44.883713+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T19:56:44.883713+0000","last_undegraded":"2024-03-16T19:56:44.883713+0000","last_fullsized":"2024-03-16T19:56:44.883713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881536+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T19:56:44.881536+0000","last_peered":"2024-03-16T19:56:44.881536+0000","last_clean":"2024-03-16T19:56:44.881536+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T19:56:44.881536+0000","last_undegraded":"2024-03-16T19:56:44.881536+0000","last_fullsized":"2024-03-16T19:56:44.881536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782932+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T19:56:44.782932+0000","last_peered":"2024-03-16T19:56:44.782932+0000","last_clean":"2024-03-16T19:56:44.782932+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T19:56:44.782932+0000","last_undegraded":"2024-03-16T19:56:44.782932+0000","last_fullsized":"2024-03-16T19:56:44.782932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880633+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T19:56:44.880633+0000","last_peered":"2024-03-16T19:56:44.880633+0000","last_clean":"2024-03-16T19:56:44.880633+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T19:56:44.880633+0000","last_undegraded":"2024-03-16T19:56:44.880633+0000","last_fullsized":"2024-03-16T19:56:44.880633+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855564+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T19:56:44.855564+0000","last_peered":"2024-03-16T19:56:44.855564+0000","last_clean":"2024-03-16T19:56:44.855564+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T19:56:44.855564+0000","last_undegraded":"2024-03-16T19:56:44.855564+0000","last_fullsized":"2024-03-16T19:56:44.855564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880999+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T19:56:44.880999+0000","last_peered":"2024-03-16T19:56:44.880999+0000","last_clean":"2024-03-16T19:56:44.880999+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T19:56:44.880999+0000","last_undegraded":"2024-03-16T19:56:44.880999+0000","last_fullsized":"2024-03-16T19:56:44.880999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881302+0000","last_change":"2024-03-16T19:56:43.461007+0000","last_active":"2024-03-16T19:56:44.881302+0000","last_peered":"2024-03-16T19:56:44.881302+0000","last_clean":"2024-03-16T19:56:44.881302+0000","last_became_active":"2024-03-16T19:56:43.460288+0000","last_became_peered":"2024-03-16T19:56:43.460288+0000","last_unstale":"2024-03-16T19:56:44.881302+0000","last_undegraded":"2024-03-16T19:56:44.881302+0000","last_fullsized":"2024-03-16T19:56:44.881302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:58.005225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783384+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T19:56:44.783384+0000","last_peered":"2024-03-16T19:56:44.783384+0000","last_clean":"2024-03-16T19:56:44.783384+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T19:56:44.783384+0000","last_undegraded":"2024-03-16T19:56:44.783384+0000","last_fullsized":"2024-03-16T19:56:44.783384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.385154+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T19:57:18.385154+0000","last_peered":"2024-03-16T19:57:18.385154+0000","last_clean":"2024-03-16T19:57:18.385154+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T19:57:18.385154+0000","last_undegraded":"2024-03-16T19:57:18.385154+0000","last_fullsized":"2024-03-16T19:57:18.385154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880798+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T19:56:44.880798+0000","last_peered":"2024-03-16T19:56:44.880798+0000","last_clean":"2024-03-16T19:56:44.880798+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T19:56:44.880798+0000","last_undegraded":"2024-03-16T19:56:44.880798+0000","last_fullsized":"2024-03-16T19:56:44.880798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881333+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T19:56:44.881333+0000","last_peered":"2024-03-16T19:56:44.881333+0000","last_clean":"2024-03-16T19:56:44.881333+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T19:56:44.881333+0000","last_undegraded":"2024-03-16T19:56:44.881333+0000","last_fullsized":"2024-03-16T19:56:44.881333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880174+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T19:56:44.880174+0000","last_peered":"2024-03-16T19:56:44.880174+0000","last_clean":"2024-03-16T19:56:44.880174+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T19:56:44.880174+0000","last_undegraded":"2024-03-16T19:56:44.880174+0000","last_fullsized":"2024-03-16T19:56:44.880174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.585324+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T19:57:15.585324+0000","last_peered":"2024-03-16T19:57:15.585324+0000","last_clean":"2024-03-16T19:57:15.585324+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T19:57:15.585324+0000","last_undegraded":"2024-03-16T19:57:15.585324+0000","last_fullsized":"2024-03-16T19:57:15.585324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869247+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T19:56:44.869247+0000","last_peered":"2024-03-16T19:56:44.869247+0000","last_clean":"2024-03-16T19:56:44.869247+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T19:56:44.869247+0000","last_undegraded":"2024-03-16T19:56:44.869247+0000","last_fullsized":"2024-03-16T19:56:44.869247+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785098+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T19:56:44.785098+0000","last_peered":"2024-03-16T19:56:44.785098+0000","last_clean":"2024-03-16T19:56:44.785098+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T19:56:44.785098+0000","last_undegraded":"2024-03-16T19:56:44.785098+0000","last_fullsized":"2024-03-16T19:56:44.785098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879070+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T19:56:44.879070+0000","last_peered":"2024-03-16T19:56:44.879070+0000","last_clean":"2024-03-16T19:56:44.879070+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T19:56:44.879070+0000","last_undegraded":"2024-03-16T19:56:44.879070+0000","last_fullsized":"2024-03-16T19:56:44.879070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879660+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T19:56:44.879660+0000","last_peered":"2024-03-16T19:56:44.879660+0000","last_clean":"2024-03-16T19:56:44.879660+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T19:56:44.879660+0000","last_undegraded":"2024-03-16T19:56:44.879660+0000","last_fullsized":"2024-03-16T19:56:44.879660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883871+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T19:56:44.883871+0000","last_peered":"2024-03-16T19:56:44.883871+0000","last_clean":"2024-03-16T19:56:44.883871+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T19:56:44.883871+0000","last_undegraded":"2024-03-16T19:56:44.883871+0000","last_fullsized":"2024-03-16T19:56:44.883871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881090+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T19:56:44.881090+0000","last_peered":"2024-03-16T19:56:44.881090+0000","last_clean":"2024-03-16T19:56:44.881090+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T19:56:44.881090+0000","last_undegraded":"2024-03-16T19:56:44.881090+0000","last_fullsized":"2024-03-16T19:56:44.881090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784903+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T19:56:44.784903+0000","last_peered":"2024-03-16T19:56:44.784903+0000","last_clean":"2024-03-16T19:56:44.784903+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T19:56:44.784903+0000","last_undegraded":"2024-03-16T19:56:44.784903+0000","last_fullsized":"2024-03-16T19:56:44.784903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856454+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T19:56:44.856454+0000","last_peered":"2024-03-16T19:56:44.856454+0000","last_clean":"2024-03-16T19:56:44.856454+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T19:56:44.856454+0000","last_undegraded":"2024-03-16T19:56:44.856454+0000","last_fullsized":"2024-03-16T19:56:44.856454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883722+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T19:56:44.883722+0000","last_peered":"2024-03-16T19:56:44.883722+0000","last_clean":"2024-03-16T19:56:44.883722+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T19:56:44.883722+0000","last_undegraded":"2024-03-16T19:56:44.883722+0000","last_fullsized":"2024-03-16T19:56:44.883722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784862+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T19:56:44.784862+0000","last_peered":"2024-03-16T19:56:44.784862+0000","last_clean":"2024-03-16T19:56:44.784862+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T19:56:44.784862+0000","last_undegraded":"2024-03-16T19:56:44.784862+0000","last_fullsized":"2024-03-16T19:56:44.784862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880645+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T19:56:44.880645+0000","last_peered":"2024-03-16T19:56:44.880645+0000","last_clean":"2024-03-16T19:56:44.880645+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T19:56:44.880645+0000","last_undegraded":"2024-03-16T19:56:44.880645+0000","last_fullsized":"2024-03-16T19:56:44.880645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783243+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T19:56:44.783243+0000","last_peered":"2024-03-16T19:56:44.783243+0000","last_clean":"2024-03-16T19:56:44.783243+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T19:56:44.783243+0000","last_undegraded":"2024-03-16T19:56:44.783243+0000","last_fullsized":"2024-03-16T19:56:44.783243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883489+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T19:56:44.883489+0000","last_peered":"2024-03-16T19:56:44.883489+0000","last_clean":"2024-03-16T19:56:44.883489+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T19:56:44.883489+0000","last_undegraded":"2024-03-16T19:56:44.883489+0000","last_fullsized":"2024-03-16T19:56:44.883489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880427+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T19:56:44.880427+0000","last_peered":"2024-03-16T19:56:44.880427+0000","last_clean":"2024-03-16T19:56:44.880427+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T19:56:44.880427+0000","last_undegraded":"2024-03-16T19:56:44.880427+0000","last_fullsized":"2024-03-16T19:56:44.880427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881350+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T19:56:44.881350+0000","last_peered":"2024-03-16T19:56:44.881350+0000","last_clean":"2024-03-16T19:56:44.881350+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T19:56:44.881350+0000","last_undegraded":"2024-03-16T19:56:44.881350+0000","last_fullsized":"2024-03-16T19:56:44.881350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885426+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T19:56:44.885426+0000","last_peered":"2024-03-16T19:56:44.885426+0000","last_clean":"2024-03-16T19:56:44.885426+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T19:56:44.885426+0000","last_undegraded":"2024-03-16T19:56:44.885426+0000","last_fullsized":"2024-03-16T19:56:44.885426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880885+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T19:56:44.880885+0000","last_peered":"2024-03-16T19:56:44.880885+0000","last_clean":"2024-03-16T19:56:44.880885+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T19:56:44.880885+0000","last_undegraded":"2024-03-16T19:56:44.880885+0000","last_fullsized":"2024-03-16T19:56:44.880885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885871+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T19:56:44.885871+0000","last_peered":"2024-03-16T19:56:44.885871+0000","last_clean":"2024-03-16T19:56:44.885871+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T19:56:44.885871+0000","last_undegraded":"2024-03-16T19:56:44.885871+0000","last_fullsized":"2024-03-16T19:56:44.885871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880278+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T19:56:44.880278+0000","last_peered":"2024-03-16T19:56:44.880278+0000","last_clean":"2024-03-16T19:56:44.880278+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T19:56:44.880278+0000","last_undegraded":"2024-03-16T19:56:44.880278+0000","last_fullsized":"2024-03-16T19:56:44.880278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855681+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T19:56:44.855681+0000","last_peered":"2024-03-16T19:56:44.855681+0000","last_clean":"2024-03-16T19:56:44.855681+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T19:56:44.855681+0000","last_undegraded":"2024-03-16T19:56:44.855681+0000","last_fullsized":"2024-03-16T19:56:44.855681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881487+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T19:56:44.881487+0000","last_peered":"2024-03-16T19:56:44.881487+0000","last_clean":"2024-03-16T19:56:44.881487+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T19:56:44.881487+0000","last_undegraded":"2024-03-16T19:56:44.881487+0000","last_fullsized":"2024-03-16T19:56:44.881487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783246+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T19:56:44.783246+0000","last_peered":"2024-03-16T19:56:44.783246+0000","last_clean":"2024-03-16T19:56:44.783246+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T19:56:44.783246+0000","last_undegraded":"2024-03-16T19:56:44.783246+0000","last_fullsized":"2024-03-16T19:56:44.783246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871059+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T19:56:44.871059+0000","last_peered":"2024-03-16T19:56:44.871059+0000","last_clean":"2024-03-16T19:56:44.871059+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T19:56:44.871059+0000","last_undegraded":"2024-03-16T19:56:44.871059+0000","last_fullsized":"2024-03-16T19:56:44.871059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882578+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T19:56:44.882578+0000","last_peered":"2024-03-16T19:56:44.882578+0000","last_clean":"2024-03-16T19:56:44.882578+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T19:56:44.882578+0000","last_undegraded":"2024-03-16T19:56:44.882578+0000","last_fullsized":"2024-03-16T19:56:44.882578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883483+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T19:56:44.883483+0000","last_peered":"2024-03-16T19:56:44.883483+0000","last_clean":"2024-03-16T19:56:44.883483+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T19:56:44.883483+0000","last_undegraded":"2024-03-16T19:56:44.883483+0000","last_fullsized":"2024-03-16T19:56:44.883483+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887376+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T19:56:44.887376+0000","last_peered":"2024-03-16T19:56:44.887376+0000","last_clean":"2024-03-16T19:56:44.887376+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T19:56:44.887376+0000","last_undegraded":"2024-03-16T19:56:44.887376+0000","last_fullsized":"2024-03-16T19:56:44.887376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784039+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T19:56:44.784039+0000","last_peered":"2024-03-16T19:56:44.784039+0000","last_clean":"2024-03-16T19:56:44.784039+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T19:56:44.784039+0000","last_undegraded":"2024-03-16T19:56:44.784039+0000","last_fullsized":"2024-03-16T19:56:44.784039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880860+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T19:56:44.880860+0000","last_peered":"2024-03-16T19:56:44.880860+0000","last_clean":"2024-03-16T19:56:44.880860+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T19:56:44.880860+0000","last_undegraded":"2024-03-16T19:56:44.880860+0000","last_fullsized":"2024-03-16T19:56:44.880860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883261+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T19:56:44.883261+0000","last_peered":"2024-03-16T19:56:44.883261+0000","last_clean":"2024-03-16T19:56:44.883261+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T19:56:44.883261+0000","last_undegraded":"2024-03-16T19:56:44.883261+0000","last_fullsized":"2024-03-16T19:56:44.883261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882289+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T19:56:44.882289+0000","last_peered":"2024-03-16T19:56:44.882289+0000","last_clean":"2024-03-16T19:56:44.882289+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T19:56:44.882289+0000","last_undegraded":"2024-03-16T19:56:44.882289+0000","last_fullsized":"2024-03-16T19:56:44.882289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883418+0000","last_change":"2024-03-16T19:56:36.306936+0000","last_active":"2024-03-16T19:56:44.883418+0000","last_peered":"2024-03-16T19:56:44.883418+0000","last_clean":"2024-03-16T19:56:44.883418+0000","last_became_active":"2024-03-16T19:56:36.306436+0000","last_became_peered":"2024-03-16T19:56:36.306436+0000","last_unstale":"2024-03-16T19:56:44.883418+0000","last_undegraded":"2024-03-16T19:56:44.883418+0000","last_fullsized":"2024-03-16T19:56:44.883418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879751+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T19:56:44.879751+0000","last_peered":"2024-03-16T19:56:44.879751+0000","last_clean":"2024-03-16T19:56:44.879751+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T19:56:44.879751+0000","last_undegraded":"2024-03-16T19:56:44.879751+0000","last_fullsized":"2024-03-16T19:56:44.879751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856790+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T19:56:44.856790+0000","last_peered":"2024-03-16T19:56:44.856790+0000","last_clean":"2024-03-16T19:56:44.856790+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T19:56:44.856790+0000","last_undegraded":"2024-03-16T19:56:44.856790+0000","last_fullsized":"2024-03-16T19:56:44.856790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887093+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T19:56:44.887093+0000","last_peered":"2024-03-16T19:56:44.887093+0000","last_clean":"2024-03-16T19:56:44.887093+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T19:56:44.887093+0000","last_undegraded":"2024-03-16T19:56:44.887093+0000","last_fullsized":"2024-03-16T19:56:44.887093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5262,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":9924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476756,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":901120,"data_stored":714188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476758,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":127080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476757,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":123328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476755,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":123364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":323584,"data_stored":127702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":123238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509461,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":913408,"data_stored":714266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":311296,"data_stored":123398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:57:19.603 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T19:57:19.603 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T19:57:19.604 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-16T19:57:19.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-16T19:57:19.604 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-16T19:57:19.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 Processor -- start 2024-03-16T19:57:19.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 -- start start 2024-03-16T19:57:19.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c3e30 0x7f1428150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14280c4210 0x7f1428151050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1428072ed0 0x7f14280c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14280c7ca0 con 0x7f14280c4210 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14280c3940 con 0x7f1428072ed0 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142f92c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1428072cc0 con 0x7f14280c3e30 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142cec7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14280c4210 0x7f1428151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142cec7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14280c4210 0x7f1428151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59458/0 (socket says 172.21.15.32:59458) 2024-03-16T19:57:19.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.796+0000 7f142cec7640 1 -- 172.21.15.32:0/2875117852 learned_addr learned my addr 172.21.15.32:0/2875117852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:19.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.800+0000 7f142cec7640 1 -- 172.21.15.32:0/2875117852 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c3e30 msgr2=0x7f1428150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:19.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.800+0000 7f142cec7640 1 --2- 172.21.15.32:0/2875117852 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c3e30 0x7f1428150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.800+0000 7f142cec7640 1 -- 172.21.15.32:0/2875117852 >> v2:172.21.15.59:3300/0 conn(0x7f1428072ed0 msgr2=0x7f14280c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:19.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.800+0000 7f142cec7640 1 --2- 172.21.15.32:0/2875117852 >> v2:172.21.15.59:3300/0 conn(0x7f1428072ed0 0x7f14280c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.800+0000 7f142cec7640 1 -- 172.21.15.32:0/2875117852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14280c7780 con 0x7f14280c4210 2024-03-16T19:57:19.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.804+0000 7f142cec7640 1 --2- 172.21.15.32:0/2875117852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280c4210 0x7f1428151050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1428157de0 tx=0x7f141801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326457c569ec2694 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f141ffff640 1 -- 172.21.15.32:0/2875117852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14180ac630 con 0x7f14280c4210 2024-03-16T19:57:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f141ffff640 1 -- 172.21.15.32:0/2875117852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14180ac790 con 0x7f14280c4210 2024-03-16T19:57:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f141ffff640 1 -- 172.21.15.32:0/2875117852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14180acb20 con 0x7f14280c4210 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- 172.21.15.32:0/2875117852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280c4210 msgr2=0x7f1428151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- 172.21.15.32:0/2875117852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280c4210 0x7f1428151050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1428157de0 tx=0x7f141801c520 comp rx=0 tx=0).stop 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- 172.21.15.32:0/2875117852 shutdown_connections 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- 172.21.15.32:0/2875117852 >> v2:172.21.15.59:3300/0 conn(0x7f1428072ed0 0x7f14280c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- 172.21.15.32:0/2875117852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280c4210 0x7f1428151050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- 172.21.15.32:0/2875117852 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c3e30 0x7f1428150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- 172.21.15.32:0/2875117852 >> 172.21.15.32:0/2875117852 conn(0x7f142806e6c0 msgr2=0x7f142806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- 172.21.15.32:0/2875117852 shutdown_connections 2024-03-16T19:57:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- 172.21.15.32:0/2875117852 wait complete. 2024-03-16T19:57:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 Processor -- start 2024-03-16T19:57:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- start start 2024-03-16T19:57:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280c3e30 0x7f14280cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c4210 0x7f14281447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428161110 con 0x7f1428072ed0 2024-03-16T19:57:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428160540 con 0x7f14280c3e30 2024-03-16T19:57:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.808+0000 7f142f92c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14281607e0 con 0x7f14280c4210 2024-03-16T19:57:19.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.812+0000 7f142d6c8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:19.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.812+0000 7f142d6c8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59462/0 (socket says 172.21.15.32:59462) 2024-03-16T19:57:19.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.812+0000 7f142d6c8640 1 -- 172.21.15.32:0/3253608127 learned_addr learned my addr 172.21.15.32:0/3253608127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:19.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 -- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c4210 msgr2=0x7f14281447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c4210 0x7f14281447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 -- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280c3e30 msgr2=0x7f14280cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:19.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280c3e30 0x7f14280cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:19.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 -- 172.21.15.32:0/3253608127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180094d0 con 0x7f1428072ed0 2024-03-16T19:57:19.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.816+0000 7f142d6c8640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f142400b550 tx=0x7f142400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:19.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.820+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424010870 con 0x7f1428072ed0 2024-03-16T19:57:19.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.820+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428160ac0 con 0x7f1428072ed0 2024-03-16T19:57:19.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.820+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142806c490 con 0x7f1428072ed0 2024-03-16T19:57:19.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.820+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1428068000 con 0x7f1428072ed0 2024-03-16T19:57:19.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.824+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14240109d0 con 0x7f1428072ed0 2024-03-16T19:57:19.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:19.824+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424010d60 con 0x7f1428072ed0 2024-03-16T19:57:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.112+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1424023480 con 0x7f1428072ed0 2024-03-16T19:57:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.112+0000 7f141dffb640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 0x7f140c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.112+0000 7f142cec7640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 0x7f140c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.112+0000 7f142cec7640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 0x7f140c069330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f14180ae000 tx=0x7f1418004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:20.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.152+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1424056130 con 0x7f1428072ed0 2024-03-16T19:57:20.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.152+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1424055800 con 0x7f1428072ed0 2024-03-16T19:57:20.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.352+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1428142d50 con 0x7f1428072ed0 2024-03-16T19:57:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.564+0000 7f141dffb640 1 -- 172.21.15.32:0/3253608127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+165195 (secure 0 0 0) 0x7f142401e090 con 0x7f1428072ed0 2024-03-16T19:57:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 msgr2=0x7f140c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 0x7f140c069330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f14180ae000 tx=0x7f1418004740 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 msgr2=0x7f142813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f142400b550 tx=0x7f142400ba10 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 shutdown_connections 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f140c066e70 0x7f140c069330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14280c4210 0x7f14281447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280c3e30 0x7f14280cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 --2- 172.21.15.32:0/3253608127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1428072ed0 0x7f142813f6e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 >> 172.21.15.32:0/3253608127 conn(0x7f142806e6c0 msgr2=0x7f142814f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 shutdown_connections 2024-03-16T19:57:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.568+0000 7f142f92c640 1 -- 172.21.15.32:0/3253608127 wait complete. 2024-03-16T19:57:20.555 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1649113964},{"type":"v1","addr":"172.21.15.32:6801","nonce":1649113964}]},"active_addr":"172.21.15.32:6801/1649113964","active_change":"2024-03-16T19:55:37.973201+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3036515665}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3285950238}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1911404268}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1306618352}]}]} 2024-03-16T19:57:20.558 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-16T19:57:20.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-16T19:57:20.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.696+0000 7f72b58cd640 1 Processor -- start 2024-03-16T19:57:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 -- start start 2024-03-16T19:57:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72b0154030 0x7f72b0072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b006b3f0 con 0x7f72b00c9de0 2024-03-16T19:57:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b006b990 con 0x7f72b0154030 2024-03-16T19:57:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72b58cd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b006b6b0 con 0x7f72b0154410 2024-03-16T19:57:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43668/0 (socket says 172.21.15.32:43668) 2024-03-16T19:57:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.700+0000 7f72af7fe640 1 -- 172.21.15.32:0/1521727095 learned_addr learned my addr 172.21.15.32:0/1521727095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.708+0000 7f72af7fe640 1 -- 172.21.15.32:0/1521727095 >> v2:172.21.15.32:3300/0 conn(0x7f72b0154030 msgr2=0x7f72b0072ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.708+0000 7f72af7fe640 1 --2- 172.21.15.32:0/1521727095 >> v2:172.21.15.32:3300/0 conn(0x7f72b0154030 0x7f72b0072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.708+0000 7f72af7fe640 1 -- 172.21.15.32:0/1521727095 >> v1:172.21.15.32:6789/0 conn(0x7f72b00c9de0 legacy=0x7f72b0159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.708+0000 7f72af7fe640 1 -- 172.21.15.32:0/1521727095 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00c9300 con 0x7f72b0154410 2024-03-16T19:57:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.720+0000 7f72af7fe640 1 --2- 172.21.15.32:0/1521727095 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009e50 tx=0x7f72a801f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c20d2b42d510d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72ae7fc640 1 -- 172.21.15.32:0/1521727095 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80af400 con 0x7f72b0154410 2024-03-16T19:57:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72ae7fc640 1 -- 172.21.15.32:0/1521727095 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a80ab050 con 0x7f72b0154410 2024-03-16T19:57:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72ae7fc640 1 -- 172.21.15.32:0/1521727095 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80af780 con 0x7f72b0154410 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 -- 172.21.15.32:0/1521727095 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 msgr2=0x7f72b00cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 --2- 172.21.15.32:0/1521727095 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009e50 tx=0x7f72a801f1e0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 -- 172.21.15.32:0/1521727095 shutdown_connections 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 --2- 172.21.15.32:0/1521727095 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b00cc620 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 --2- 172.21.15.32:0/1521727095 >> v2:172.21.15.32:3300/0 conn(0x7f72b0154030 0x7f72b0072ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 -- 172.21.15.32:0/1521727095 >> 172.21.15.32:0/1521727095 conn(0x7f72b006e6c0 msgr2=0x7f72b006c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 -- 172.21.15.32:0/1521727095 shutdown_connections 2024-03-16T19:57:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.732+0000 7f72b58cd640 1 -- 172.21.15.32:0/1521727095 wait complete. 2024-03-16T19:57:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 Processor -- start 2024-03-16T19:57:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 -- start start 2024-03-16T19:57:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 0x7f72b0146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b0154030 0x7f72b01410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b0141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01561e0 con 0x7f72b0154030 2024-03-16T19:57:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0153080 con 0x7f72b00c9de0 2024-03-16T19:57:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.736+0000 7f72b58cd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0156410 con 0x7f72b0154410 2024-03-16T19:57:20.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.748+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b0141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:20.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.748+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b0141630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43678/0 (socket says 172.21.15.32:43678) 2024-03-16T19:57:20.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.748+0000 7f72af7fe640 1 -- 172.21.15.32:0/187942934 learned_addr learned my addr 172.21.15.32:0/187942934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.756+0000 7f72aeffd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 0x7f72b0146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.768+0000 7f72aeffd640 1 -- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 msgr2=0x7f72b0141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.768+0000 7f72aeffd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b0141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.768+0000 7f72aeffd640 1 -- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b0154030 msgr2=0x7f72b01410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.768+0000 7f72aeffd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b0154030 0x7f72b01410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.768+0000 7f72aeffd640 1 -- 172.21.15.32:0/187942934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a8009b30 con 0x7f72b00c9de0 2024-03-16T19:57:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.776+0000 7f72aeffd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 0x7f72b0146720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b420 tx=0x7f72a000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a0011830 con 0x7f72b00c9de0 2024-03-16T19:57:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a0011990 con 0x7f72b00c9de0 2024-03-16T19:57:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a0011d40 con 0x7f72b00c9de0 2024-03-16T19:57:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b01566f0 con 0x7f72b00c9de0 2024-03-16T19:57:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b01535b0 con 0x7f72b00c9de0 2024-03-16T19:57:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.868+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b01537d0 con 0x7f72b00c9de0 2024-03-16T19:57:20.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.924+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f72a001b460 con 0x7f72b00c9de0 2024-03-16T19:57:20.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.924+0000 7f72acff9640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:20.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.924+0000 7f72a7fff640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:20.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:20.924+0000 7f72a7fff640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72b01529a0 tx=0x7f72980079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:21.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.200+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f72a0056520 con 0x7f72b00c9de0 2024-03-16T19:57:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.208+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f72a00520c0 con 0x7f72b00c9de0 2024-03-16T19:57:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.388+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72b0068000 con 0x7f72b00c9de0 2024-03-16T19:57:21.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.456+0000 7f72acff9640 1 -- 172.21.15.32:0/187942934 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f72a0055780 con 0x7f72b00c9de0 2024-03-16T19:57:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 msgr2=0x7f7290069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72b01529a0 tx=0x7f72980079c0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 msgr2=0x7f72b0146720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 0x7f72b0146720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b420 tx=0x7f72a000b8e0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 shutdown_connections 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72b0154410 0x7f72b0141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72b0154030 0x7f72b01410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 --2- 172.21.15.32:0/187942934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b00c9de0 0x7f72b0146720 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 >> 172.21.15.32:0/187942934 conn(0x7f72b006e6c0 msgr2=0x7f72b01e6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:21.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 shutdown_connections 2024-03-16T19:57:21.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.460+0000 7f72b58cd640 1 -- 172.21.15.32:0/187942934 wait complete. 2024-03-16T19:57:21.452 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-16T19:57:21.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.596+0000 7fb7d3cdb640 1 Processor -- start 2024-03-16T19:57:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.596+0000 7fb7d3cdb640 1 -- start start 2024-03-16T19:57:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d3cdb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc151220 0x7fb7cc0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d3cdb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7cc0c4210 0x7fb7cc0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d3cdb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0c3940 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d3cdb640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c7ca0 con 0x7fb7cc0c4210 2024-03-16T19:57:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d3cdb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc072cc0 con 0x7fb7cc151220 2024-03-16T19:57:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d2278640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7cc0c4210 0x7fb7cc0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d1a77640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb7cc0c3e30 0x7fb7cc150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48874/0 (socket says 172.21.15.32:48874) 2024-03-16T19:57:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d1a77640 1 -- 172.21.15.32:0/1326395601 learned_addr learned my addr 172.21.15.32:0/1326395601 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.600+0000 7fb7d1276640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc151220 0x7fb7cc0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.604+0000 7fb7d2278640 1 -- 172.21.15.32:0/1326395601 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc151220 msgr2=0x7fb7cc0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:21.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.604+0000 7fb7d2278640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc151220 0x7fb7cc0c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.604+0000 7fb7d2278640 1 -- 172.21.15.32:0/1326395601 >> v1:172.21.15.32:6789/0 conn(0x7fb7cc0c3e30 legacy=0x7fb7cc150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.604+0000 7fb7d2278640 1 -- 172.21.15.32:0/1326395601 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0c7780 con 0x7fb7cc0c4210 2024-03-16T19:57:21.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.608+0000 7fb7d2278640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c4210 0x7fb7cc0c8630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009dc0 tx=0x7fb7c401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a496cf663fbbdb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d0a75640 1 -- 172.21.15.32:0/1326395601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40a5900 con 0x7fb7cc0c4210 2024-03-16T19:57:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d0a75640 1 -- 172.21.15.32:0/1326395601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c40a5a60 con 0x7fb7cc0c4210 2024-03-16T19:57:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d0a75640 1 -- 172.21.15.32:0/1326395601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40a5df0 con 0x7fb7cc0c4210 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1326395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c4210 msgr2=0x7fb7cc0c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c4210 0x7fb7cc0c8630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009dc0 tx=0x7fb7c401c950 comp rx=0 tx=0).stop 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1326395601 shutdown_connections 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c4210 0x7fb7cc0c8630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1326395601 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc151220 0x7fb7cc0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1326395601 >> 172.21.15.32:0/1326395601 conn(0x7fb7cc06e6c0 msgr2=0x7fb7cc06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1326395601 shutdown_connections 2024-03-16T19:57:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.612+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1326395601 wait complete. 2024-03-16T19:57:21.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 Processor -- start 2024-03-16T19:57:21.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 -- start start 2024-03-16T19:57:21.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc0c4210 0x7fb7cc144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7cc151220 0x7fb7cc145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc1f27c0 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc1f2420 con 0x7fb7cc151220 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d3cdb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0cc5f0 con 0x7fb7cc0c4210 2024-03-16T19:57:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d1a77640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d1a77640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59494/0 (socket says 172.21.15.32:59494) 2024-03-16T19:57:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d1a77640 1 -- 172.21.15.32:0/1350360375 learned_addr learned my addr 172.21.15.32:0/1350360375 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d1276640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc0c4210 0x7fb7cc144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.616+0000 7fb7d2278640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7cc151220 0x7fb7cc145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.620+0000 7fb7d1a77640 1 -- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc0c4210 msgr2=0x7fb7cc144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.620+0000 7fb7d1a77640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc0c4210 0x7fb7cc144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.620+0000 7fb7d1a77640 1 -- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7cc151220 msgr2=0x7fb7cc145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.620+0000 7fb7d1a77640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7cc151220 0x7fb7cc145510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.620+0000 7fb7d1a77640 1 -- 172.21.15.32:0/1350360375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4009a70 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.624+0000 7fb7d1a77640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800c420 tx=0x7fb7c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8005cf0 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c8005e50 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc0cc8b0 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80072f0 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc0cbef0 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.628+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc1f2280 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.928+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb7c8007450 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.928+0000 7fb7c27fc640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 0x7fb7b0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.928+0000 7fb7d1276640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 0x7fb7b0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.932+0000 7fb7d1276640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 0x7fb7b0069350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc157930 tx=0x7fb7bc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.948+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb7c8054e80 con 0x7fb7cc0c3e30 2024-03-16T19:57:21.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:21.948+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb7c8051120 con 0x7fb7cc0c3e30 2024-03-16T19:57:22.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.136+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7cc1f2590 con 0x7fb7cc0c3e30 2024-03-16T19:57:22.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.224+0000 7fb7c27fc640 1 -- 172.21.15.32:0/1350360375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fb7cc1f2590 con 0x7fb7cc0c3e30 2024-03-16T19:57:22.196 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:22.196 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 msgr2=0x7fb7b0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 0x7fb7b0069350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc157930 tx=0x7fb7bc009190 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 msgr2=0x7fb7cc1f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800c420 tx=0x7fb7c800c8e0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 shutdown_connections 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb7b0066e90 0x7fb7b0069350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7cc151220 0x7fb7cc145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7cc0c4210 0x7fb7cc144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 --2- 172.21.15.32:0/1350360375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7cc0c3e30 0x7fb7cc1f2be0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 >> 172.21.15.32:0/1350360375 conn(0x7fb7cc06e6c0 msgr2=0x7fb7cc070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 shutdown_connections 2024-03-16T19:57:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.228+0000 7fb7d3cdb640 1 -- 172.21.15.32:0/1350360375 wait complete. 2024-03-16T19:57:22.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-16T19:57:22.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-16T19:57:22.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-16T19:57:22.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-16T19:57:22.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-16T19:57:22.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-16T19:57:22.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-16T19:57:22.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-16T19:57:22.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 Processor -- start 2024-03-16T19:57:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 -- start start 2024-03-16T19:57:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85b814fee0 0x7f85b81568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b80c3c00 con 0x7f85b80c6670 2024-03-16T19:57:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b8156e20 con 0x7f85b80c4090 2024-03-16T19:57:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b7fff640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b81597e0 con 0x7f85b814fee0 2024-03-16T19:57:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b67fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f85b80c6670 0x7f85b806d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48884/0 (socket says 172.21.15.32:48884) 2024-03-16T19:57:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.468+0000 7f85b67fc640 1 -- 172.21.15.32:0/1955704320 learned_addr learned my addr 172.21.15.32:0/1955704320 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f85b6ffd640 1 --2- 172.21.15.32:0/1955704320 >> v2:172.21.15.59:3300/0 conn(0x7f85b814fee0 0x7f85b81568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 Processor -- start 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 -- start start 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a540c68e0 0x7f3a540c8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 0x7f3a54160960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a54156a60 con 0x7f3a5414caa0 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5414ce80 con 0x7f3a540c87b0 2024-03-16T19:57:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f3a5b2be640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a54156c90 con 0x7f3a540c68e0 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 Processor -- start 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 -- start start 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff58406ce10 0x7ff5840c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 0x7ff5841640e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff584164620 con 0x7ff58406d1f0 2024-03-16T19:57:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5841678d0 con 0x7ff58406ce10 2024-03-16T19:57:22.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff583fff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840c6980 con 0x7ff584163cb0 2024-03-16T19:57:22.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff5837fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 0x7ff5841640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff5827fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff58406d1f0 0x7ff5840c6270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48886/0 (socket says 172.21.15.32:48886) 2024-03-16T19:57:22.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7ff5827fc640 1 -- 172.21.15.32:0/2173078220 learned_addr learned my addr 172.21.15.32:0/2173078220 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 Processor -- start 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 -- start start 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f006e460 con 0x7f16f0164390 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00c3280 con 0x7f16f0166aa0 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f16f5a50640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f006e7b0 con 0x7f16f0163fb0 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f9465ccc640 1 Processor -- start 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.472+0000 7f9465ccc640 1 -- start start 2024-03-16T19:57:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7ff582ffd640 1 --2- 172.21.15.32:0/2173078220 >> v2:172.21.15.32:3300/0 conn(0x7ff58406ce10 0x7ff5840c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7f9465ccc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9460163d50 0x7f94601641a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7f9465ccc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946006e460 con 0x7f94600c73b0 2024-03-16T19:57:22.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7f9465ccc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9460166fe0 con 0x7f9460166aa0 2024-03-16T19:57:22.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7f9465ccc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946006e7b0 con 0x7f9460163d50 2024-03-16T19:57:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7ff5837fe640 1 -- 172.21.15.32:0/2173078220 >> v2:172.21.15.32:3300/0 conn(0x7ff58406ce10 msgr2=0x7ff5840c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7ff5837fe640 1 --2- 172.21.15.32:0/2173078220 >> v2:172.21.15.32:3300/0 conn(0x7ff58406ce10 0x7ff5840c7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7ff5837fe640 1 -- 172.21.15.32:0/2173078220 >> v1:172.21.15.32:6789/0 conn(0x7ff58406d1f0 legacy=0x7ff5840c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.476+0000 7ff5837fe640 1 -- 172.21.15.32:0/2173078220 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff584166c90 con 0x7ff584163cb0 2024-03-16T19:57:22.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f85b6ffd640 1 -- 172.21.15.32:0/1955704320 >> v1:172.21.15.59:6789/0 conn(0x7f85b80c4090 legacy=0x7f85b80c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f85b6ffd640 1 -- 172.21.15.32:0/1955704320 >> v1:172.21.15.32:6789/0 conn(0x7f85b80c6670 legacy=0x7f85b806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f85b6ffd640 1 -- 172.21.15.32:0/1955704320 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b8159ac0 con 0x7f85b814fee0 2024-03-16T19:57:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f16eeffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f16eeffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43706/0 (socket says 172.21.15.32:43706) 2024-03-16T19:57:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f16eeffd640 1 -- 172.21.15.32:0/2913492712 learned_addr learned my addr 172.21.15.32:0/2913492712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.480+0000 7f3a5985b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 0x7f3a54160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.484+0000 7f3a5985b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 0x7f3a54160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59504/0 (socket says 172.21.15.32:59504) 2024-03-16T19:57:22.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.484+0000 7f3a5985b640 1 -- 172.21.15.32:0/183322716 learned_addr learned my addr 172.21.15.32:0/183322716 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.484+0000 7ff5837fe640 1 --2- 172.21.15.32:0/2173078220 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 0x7ff5841640e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b370 tx=0x7ff57401f590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08223407daaa254 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.484+0000 7f945effd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9460163d50 0x7f94601641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f16eeffd640 1 -- 172.21.15.32:0/2913492712 >> v1:172.21.15.59:6789/0 conn(0x7f16f0166aa0 legacy=0x7f16f006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f16eeffd640 1 -- 172.21.15.32:0/2913492712 >> v1:172.21.15.32:6789/0 conn(0x7f16f0164390 legacy=0x7f16f00c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f16eeffd640 1 -- 172.21.15.32:0/2913492712 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0166fe0 con 0x7f16f0163fb0 2024-03-16T19:57:22.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f85b6ffd640 1 --2- 172.21.15.32:0/1955704320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b814fee0 0x7f85b81568e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009fd0 tx=0x7f85ac01fe80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b31f7fa3656e54b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f945effd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9460163d50 0x7f94601641a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59506/0 (socket says 172.21.15.32:59506) 2024-03-16T19:57:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.488+0000 7f945effd640 1 -- 172.21.15.32:0/583400673 learned_addr learned my addr 172.21.15.32:0/583400673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.492+0000 7f3a58859640 1 --2- 172.21.15.32:0/183322716 >> v2:172.21.15.59:3300/0 conn(0x7f3a540c68e0 0x7f3a540c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7f945effd640 1 -- 172.21.15.32:0/583400673 >> v1:172.21.15.59:6789/0 conn(0x7f9460166aa0 legacy=0x7f946006ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7f945effd640 1 -- 172.21.15.32:0/583400673 >> v1:172.21.15.32:6789/0 conn(0x7f94600c73b0 legacy=0x7f946006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7f945effd640 1 -- 172.21.15.32:0/583400673 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94601672c0 con 0x7f9460163d50 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7ff581ffb640 1 -- 172.21.15.32:0/2173078220 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740af600 con 0x7ff584163cb0 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7ff581ffb640 1 -- 172.21.15.32:0/2173078220 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5740af760 con 0x7ff584163cb0 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.496+0000 7ff581ffb640 1 -- 172.21.15.32:0/2173078220 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740afaf0 con 0x7ff584163cb0 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7f3a58859640 1 -- 172.21.15.32:0/183322716 >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 msgr2=0x7f3a54160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7f3a58859640 1 --2- 172.21.15.32:0/183322716 >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 0x7f3a54160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7f3a58859640 1 -- 172.21.15.32:0/183322716 >> v1:172.21.15.32:6789/0 conn(0x7f3a5414caa0 legacy=0x7f3a54159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7f3a58859640 1 -- 172.21.15.32:0/183322716 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54156f30 con 0x7f3a540c68e0 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- 172.21.15.32:0/2173078220 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 msgr2=0x7ff5841640e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- 172.21.15.32:0/2173078220 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 0x7ff5841640e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b370 tx=0x7ff57401f590 comp rx=0 tx=0).stop 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- 172.21.15.32:0/2173078220 shutdown_connections 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- 172.21.15.32:0/2173078220 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff584163cb0 0x7ff5841640e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- 172.21.15.32:0/2173078220 >> v2:172.21.15.32:3300/0 conn(0x7ff58406ce10 0x7ff5840c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- 172.21.15.32:0/2173078220 >> 172.21.15.32:0/2173078220 conn(0x7ff58406b190 msgr2=0x7ff58406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- 172.21.15.32:0/2173078220 shutdown_connections 2024-03-16T19:57:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7f16eeffd640 1 --2- 172.21.15.32:0/2913492712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16e0008a20 tx=0x7f16e001ea10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6f65e5af5af3f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- 172.21.15.32:0/2173078220 wait complete. 2024-03-16T19:57:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 Processor -- start 2024-03-16T19:57:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- start start 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff58406ce10 0x7ff5841fe970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 0x7ff5840ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff584163cb0 0x7ff5840ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841fdf60 con 0x7ff584163cb0 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841ed1a0 con 0x7ff58406ce10 2024-03-16T19:57:22.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.500+0000 7ff583fff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841feeb0 con 0x7ff58406d1f0 2024-03-16T19:57:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 Processor -- start 2024-03-16T19:57:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 -- start start 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a5c0cc5d0 0x7f3a5c15ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c0c3870 con 0x7f3a5c158420 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c0cd040 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f3a63659640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0cca80 con 0x7f3a5c158800 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 Processor -- start 2024-03-16T19:57:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 -- start start 2024-03-16T19:57:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 0x7f076c0064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c007c20 con 0x7f076c035500 2024-03-16T19:57:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0459e0 con 0x7f076c0d5cc0 2024-03-16T19:57:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.504+0000 7f077359e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c008430 con 0x7f076c033b70 2024-03-16T19:57:22.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.508+0000 7f3a61bf6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.508+0000 7f3a58859640 1 --2- 172.21.15.32:0/183322716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 0x7f3a540c8270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a54071130 tx=0x7f3a4401ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6415440bcff93cba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.508+0000 7ff582ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff58406ce10 0x7ff5841fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.508+0000 7ff5827fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 0x7ff5840ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.512+0000 7f3a61bf6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43722/0 (socket says 172.21.15.32:43722) 2024-03-16T19:57:22.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.512+0000 7f3a61bf6640 1 -- 172.21.15.32:0/1715814569 learned_addr learned my addr 172.21.15.32:0/1715814569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.512+0000 7ff5827fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 0x7ff5840ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43710/0 (socket says 172.21.15.32:43710) 2024-03-16T19:57:22.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.512+0000 7ff5827fc640 1 -- 172.21.15.32:0/524049299 learned_addr learned my addr 172.21.15.32:0/524049299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f3a61bf6640 1 -- 172.21.15.32:0/1715814569 >> v2:172.21.15.59:3300/0 conn(0x7f3a5c0cc5d0 msgr2=0x7f3a5c15ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f3a61bf6640 1 --2- 172.21.15.32:0/1715814569 >> v2:172.21.15.59:3300/0 conn(0x7f3a5c0cc5d0 0x7f3a5c15ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f3a61bf6640 1 -- 172.21.15.32:0/1715814569 >> v1:172.21.15.32:6789/0 conn(0x7f3a5c158420 legacy=0x7f3a5c0c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f3a61bf6640 1 -- 172.21.15.32:0/1715814569 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0ccd60 con 0x7f3a5c158800 2024-03-16T19:57:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f2e88e0d640 1 Processor -- start 2024-03-16T19:57:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f2e88e0d640 1 -- start start 2024-03-16T19:57:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.516+0000 7f2e88e0d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e8415a1d0 0x7f2e840c6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7f2e88e0d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e840c5760 con 0x7f2e84072bd0 2024-03-16T19:57:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7f2e88e0d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e84151400 con 0x7f2e84151020 2024-03-16T19:57:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7f2e88e0d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e840c6ac0 con 0x7f2e8415a1d0 2024-03-16T19:57:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7f945effd640 1 --2- 172.21.15.32:0/583400673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460163d50 0x7f94601641a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9454009870 tx=0x7f945401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbe8c57eee764b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7ff5827fc640 1 -- 172.21.15.32:0/524049299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff58406ce10 msgr2=0x7ff5841fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7ff5827fc640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff58406ce10 0x7ff5841fe970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7ff5827fc640 1 -- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff584163cb0 msgr2=0x7ff5840ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7ff5827fc640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff584163cb0 0x7ff5840ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.520+0000 7ff5827fc640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57400b050 con 0x7ff58406d1f0 2024-03-16T19:57:22.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16edffb640 1 -- 172.21.15.32:0/2913492712 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e00a79d0 con 0x7f16f0163fb0 2024-03-16T19:57:22.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16edffb640 1 -- 172.21.15.32:0/2913492712 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e00ab070 con 0x7f16f0163fb0 2024-03-16T19:57:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16edffb640 1 -- 172.21.15.32:0/2913492712 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e00a7d70 con 0x7f16f0163fb0 2024-03-16T19:57:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f077259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f076c0d5cc0 0x7f076c046150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52542/0 (socket says 172.21.15.32:52542) 2024-03-16T19:57:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f077259c640 1 -- 172.21.15.32:0/3380444923 learned_addr learned my addr 172.21.15.32:0/3380444923 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 -- 172.21.15.32:0/2913492712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 msgr2=0x7f16f006ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 --2- 172.21.15.32:0/2913492712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16e0008a20 tx=0x7f16e001ea10 comp rx=0 tx=0).stop 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 -- 172.21.15.32:0/2913492712 shutdown_connections 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 --2- 172.21.15.32:0/2913492712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f006ebd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 -- 172.21.15.32:0/2913492712 >> 172.21.15.32:0/2913492712 conn(0x7f16f006b190 msgr2=0x7f16f006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 -- 172.21.15.32:0/2913492712 shutdown_connections 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.524+0000 7f16f5a50640 1 -- 172.21.15.32:0/2913492712 wait complete. 2024-03-16T19:57:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 Processor -- start 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 -- start start 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16f0164390 0x7f16f015d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 0x7f16f015d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f01ffaf0 con 0x7f16f0166aa0 2024-03-16T19:57:22.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0163860 con 0x7f16f0164390 2024-03-16T19:57:22.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16f5a50640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f01ffce0 con 0x7f16f0163fb0 2024-03-16T19:57:22.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f3a61bf6640 1 --2- 172.21.15.32:0/1715814569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009e50 tx=0x7f3a5401c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=960d1bea15153519 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.528+0000 7f16ef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 0x7f16f015d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7ff5827fc640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 0x7ff5840ca380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c500 tx=0x7ff57800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7f16ef7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 0x7f16f015d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59526/0 (socket says 172.21.15.32:59526) 2024-03-16T19:57:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7f16ef7fe640 1 -- 172.21.15.32:0/2210966094 learned_addr learned my addr 172.21.15.32:0/2210966094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7f2e81d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e8415a1d0 0x7f2e840c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7f2e81d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e8415a1d0 0x7f2e840c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59522/0 (socket says 172.21.15.32:59522) 2024-03-16T19:57:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.532+0000 7f2e81d9b640 1 -- 172.21.15.32:0/3914328142 learned_addr learned my addr 172.21.15.32:0/3914328142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.536+0000 7f16ef7fe640 1 -- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 msgr2=0x7f16f015acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.536+0000 7f16ef7fe640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16f0163fb0 0x7f16f015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.536+0000 7f16ef7fe640 1 -- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16f0164390 msgr2=0x7f16f015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.536+0000 7f16ef7fe640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16f0164390 0x7f16f015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.536+0000 7f16ef7fe640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00086d0 con 0x7f16f0166aa0 2024-03-16T19:57:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b5ffb640 1 -- 172.21.15.32:0/1955704320 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0a9c90 con 0x7f85b814fee0 2024-03-16T19:57:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b5ffb640 1 -- 172.21.15.32:0/1955704320 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0a9df0 con 0x7f85b814fee0 2024-03-16T19:57:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b5ffb640 1 -- 172.21.15.32:0/1955704320 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0b1660 con 0x7f85b814fee0 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 -- 172.21.15.32:0/1955704320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b814fee0 msgr2=0x7f85b81568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 --2- 172.21.15.32:0/1955704320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b814fee0 0x7f85b81568e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009fd0 tx=0x7f85ac01fe80 comp rx=0 tx=0).stop 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 -- 172.21.15.32:0/1955704320 shutdown_connections 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 --2- 172.21.15.32:0/1955704320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b814fee0 0x7f85b81568e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 -- 172.21.15.32:0/1955704320 >> 172.21.15.32:0/1955704320 conn(0x7f85b806b190 msgr2=0x7f85b806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f2e81d9b640 1 -- 172.21.15.32:0/3914328142 >> v1:172.21.15.59:6789/0 conn(0x7f2e84151020 legacy=0x7f2e84156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f2e81d9b640 1 -- 172.21.15.32:0/3914328142 >> v1:172.21.15.32:6789/0 conn(0x7f2e84072bd0 legacy=0x7f2e8406c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f2e81d9b640 1 -- 172.21.15.32:0/3914328142 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e840c5b10 con 0x7f2e8415a1d0 2024-03-16T19:57:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 -- 172.21.15.32:0/1955704320 shutdown_connections 2024-03-16T19:57:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.540+0000 7f85b7fff640 1 -- 172.21.15.32:0/1955704320 wait complete. 2024-03-16T19:57:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 Processor -- start 2024-03-16T19:57:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 -- start start 2024-03-16T19:57:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a4bfff640 1 -- 172.21.15.32:0/183322716 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440a88b0 con 0x7f3a540c68e0 2024-03-16T19:57:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a4bfff640 1 -- 172.21.15.32:0/183322716 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a440a8a10 con 0x7f3a540c68e0 2024-03-16T19:57:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a4bfff640 1 -- 172.21.15.32:0/183322716 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440a8da0 con 0x7f3a540c68e0 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 0x7f85b806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85b80c6670 0x7f85b81f1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85b814fee0 0x7f85b814bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b81f0de0 con 0x7f85b814fee0 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b81f0a90 con 0x7f85b80c4090 2024-03-16T19:57:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f85b7fff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8151fd0 con 0x7f85b80c6670 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- 172.21.15.32:0/183322716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 msgr2=0x7f3a540c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/183322716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 0x7f3a540c8270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a54071130 tx=0x7f3a4401ebd0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- 172.21.15.32:0/183322716 shutdown_connections 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/183322716 >> v2:172.21.15.32:3300/0 conn(0x7f3a540c87b0 0x7f3a54160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/183322716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 0x7f3a540c8270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- 172.21.15.32:0/183322716 >> 172.21.15.32:0/183322716 conn(0x7f3a5406afc0 msgr2=0x7f3a5406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- 172.21.15.32:0/183322716 shutdown_connections 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- 172.21.15.32:0/183322716 wait complete. 2024-03-16T19:57:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f16ef7fe640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 0x7f16f015d690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b410 tx=0x7f16e400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 Processor -- start 2024-03-16T19:57:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f3a5b2be640 1 -- start start 2024-03-16T19:57:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.544+0000 7f0771d9b640 1 --2- 172.21.15.32:0/3380444923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 0x7f076c0064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 0x7f3a54158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a540c87b0 0x7f3a5414a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5414caa0 0x7f3a5414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541ec6c0 con 0x7f3a540c87b0 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541eb780 con 0x7f3a540c68e0 2024-03-16T19:57:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f3a5b2be640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541eba40 con 0x7f3a5414caa0 2024-03-16T19:57:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f85b6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 0x7f85b806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f2e81d9b640 1 --2- 172.21.15.32:0/3914328142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 0x7f2e840c6580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009870 tx=0x7f2e7801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77352318a653b2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f945e7fc640 1 -- 172.21.15.32:0/583400673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540ab600 con 0x7f9460163d50 2024-03-16T19:57:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f945e7fc640 1 -- 172.21.15.32:0/583400673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94540a7050 con 0x7f9460163d50 2024-03-16T19:57:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.548+0000 7f945e7fc640 1 -- 172.21.15.32:0/583400673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540ab980 con 0x7f9460163d50 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- 172.21.15.32:0/583400673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460163d50 msgr2=0x7f94601641a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 --2- 172.21.15.32:0/583400673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460163d50 0x7f94601641a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9454009870 tx=0x7f945401c4e0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- 172.21.15.32:0/583400673 shutdown_connections 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 --2- 172.21.15.32:0/583400673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460163d50 0x7f94601641a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- 172.21.15.32:0/583400673 >> 172.21.15.32:0/583400673 conn(0x7f946006b190 msgr2=0x7f946006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- 172.21.15.32:0/583400673 shutdown_connections 2024-03-16T19:57:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- 172.21.15.32:0/583400673 wait complete. 2024-03-16T19:57:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f0771d9b640 1 -- 172.21.15.32:0/3380444923 >> v1:172.21.15.59:6789/0 conn(0x7f076c0d5cc0 legacy=0x7f076c046150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f0771d9b640 1 -- 172.21.15.32:0/3380444923 >> v1:172.21.15.32:6789/0 conn(0x7f076c035500 legacy=0x7f076c0e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f0771d9b640 1 -- 172.21.15.32:0/3380444923 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c036500 con 0x7f076c033b70 2024-03-16T19:57:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 Processor -- start 2024-03-16T19:57:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- start start 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94600c73b0 0x7f9460207a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9460163d50 0x7f9460204490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 0x7f94601589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460207fb0 con 0x7f9460166aa0 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94600c35e0 con 0x7f94600c73b0 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f9465ccc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94600c40d0 con 0x7f9460163d50 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f85b6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 0x7f85b806ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43626/0 (socket says 172.21.15.32:43626) 2024-03-16T19:57:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f85b6ffd640 1 -- 172.21.15.32:0/640350499 learned_addr learned my addr 172.21.15.32:0/640350499 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f3a53fff640 1 -- 172.21.15.32:0/1715814569 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540a5870 con 0x7f3a5c158800 2024-03-16T19:57:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f3a53fff640 1 -- 172.21.15.32:0/1715814569 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a540a59d0 con 0x7f3a5c158800 2024-03-16T19:57:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.552+0000 7f3a53fff640 1 -- 172.21.15.32:0/1715814569 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540a5d60 con 0x7f3a5c158800 2024-03-16T19:57:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- 172.21.15.32:0/1715814569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 msgr2=0x7f3a5c161810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- 172.21.15.32:0/1715814569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009e50 tx=0x7f3a5401c9f0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- 172.21.15.32:0/1715814569 shutdown_connections 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- 172.21.15.32:0/1715814569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c158800 0x7f3a5c161810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- 172.21.15.32:0/1715814569 >> v2:172.21.15.59:3300/0 conn(0x7f3a5c0cc5d0 0x7f3a5c15ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- 172.21.15.32:0/1715814569 >> 172.21.15.32:0/1715814569 conn(0x7f3a5c066ea0 msgr2=0x7f3a5c0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- 172.21.15.32:0/1715814569 shutdown_connections 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- 172.21.15.32:0/1715814569 wait complete. 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 Processor -- start 2024-03-16T19:57:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- start start 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c0cc5d0 0x7f3a5c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a5c158420 0x7f3a5c160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a5c158800 0x7f3a5c160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c148480 con 0x7f3a5c158800 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c148ce0 con 0x7f3a5c158420 2024-03-16T19:57:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.556+0000 7f3a63659640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c5930 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.560+0000 7f85b6ffd640 1 -- 172.21.15.32:0/640350499 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85b80c6670 msgr2=0x7f85b81f1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.560+0000 7f85b6ffd640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85b80c6670 0x7f85b81f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.560+0000 7f85b6ffd640 1 -- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85b814fee0 msgr2=0x7f85b814bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.560+0000 7f85b6ffd640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85b814fee0 0x7f85b814bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.560+0000 7f85b6ffd640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac009cb0 con 0x7f85b80c4090 2024-03-16T19:57:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578005cf0 con 0x7ff58406d1f0 2024-03-16T19:57:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578005e50 con 0x7ff58406d1f0 2024-03-16T19:57:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff584201f80 con 0x7ff58406d1f0 2024-03-16T19:57:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5780072f0 con 0x7ff58406d1f0 2024-03-16T19:57:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5841fe170 con 0x7ff58406d1f0 2024-03-16T19:57:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7f0771d9b640 1 --2- 172.21.15.32:0/3380444923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 0x7f076c0064d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07680097f0 tx=0x7f076801c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02908ec8b70ef76 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.564+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5841674f0 con 0x7ff58406d1f0 2024-03-16T19:57:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e8159a640 1 -- 172.21.15.32:0/3914328142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780ab600 con 0x7f2e8415a1d0 2024-03-16T19:57:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e8159a640 1 -- 172.21.15.32:0/3914328142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e780a7050 con 0x7f2e8415a1d0 2024-03-16T19:57:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e8159a640 1 -- 172.21.15.32:0/3914328142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780ab980 con 0x7f2e8415a1d0 2024-03-16T19:57:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- 172.21.15.32:0/3914328142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 msgr2=0x7f2e840c6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 --2- 172.21.15.32:0/3914328142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 0x7f2e840c6580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009870 tx=0x7f2e7801c450 comp rx=0 tx=0).stop 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f85b6ffd640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 0x7f85b806ea50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85ac01ca90 tx=0x7f85ac01cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- 172.21.15.32:0/3914328142 shutdown_connections 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 --2- 172.21.15.32:0/3914328142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 0x7f2e840c6580 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- 172.21.15.32:0/3914328142 >> 172.21.15.32:0/3914328142 conn(0x7f2e840cc7a0 msgr2=0x7f2e840ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- 172.21.15.32:0/3914328142 shutdown_connections 2024-03-16T19:57:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- 172.21.15.32:0/3914328142 wait complete. 2024-03-16T19:57:22.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4011830 con 0x7f16f0166aa0 2024-03-16T19:57:22.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e4011990 con 0x7f16f0166aa0 2024-03-16T19:57:22.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4011d40 con 0x7f16f0166aa0 2024-03-16T19:57:22.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f01fffc0 con 0x7f16f0166aa0 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f0200580 con 0x7f16f0166aa0 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 Processor -- start 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- start start 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e84072bd0 0x7f2e8414dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 0x7f2e84151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 0x7f2e84151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e84154ab0 con 0x7f2e8415a1d0 2024-03-16T19:57:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e84152290 con 0x7f2e84072bd0 2024-03-16T19:57:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f2e88e0d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e841533a0 con 0x7f2e84151020 2024-03-16T19:57:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.568+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16d0001040 con 0x7f16f0166aa0 2024-03-16T19:57:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.572+0000 7f3a58859640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a540c87b0 0x7f3a5414a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.572+0000 7f3a58859640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a540c87b0 0x7f3a5414a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59546/0 (socket says 172.21.15.32:59546) 2024-03-16T19:57:22.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.572+0000 7f3a58859640 1 -- 172.21.15.32:0/810971291 learned_addr learned my addr 172.21.15.32:0/810971291 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.572+0000 7f945ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 0x7f94601589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0ba640 con 0x7f85b80c4090 2024-03-16T19:57:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac00c040 con 0x7f85b80c4090 2024-03-16T19:57:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b80b7ea0 con 0x7f85b80c4090 2024-03-16T19:57:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0ba9a0 con 0x7f85b80c4090 2024-03-16T19:57:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b81edce0 con 0x7f85b80c4090 2024-03-16T19:57:22.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85b815a360 con 0x7f85b80c4090 2024-03-16T19:57:22.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.576+0000 7f3a5985b640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5414caa0 0x7f3a5414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077159a640 1 -- 172.21.15.32:0/3380444923 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680ab600 con 0x7f076c033b70 2024-03-16T19:57:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077159a640 1 -- 172.21.15.32:0/3380444923 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07680a7050 con 0x7f076c033b70 2024-03-16T19:57:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077159a640 1 -- 172.21.15.32:0/3380444923 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680ab980 con 0x7f076c033b70 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 -- 172.21.15.32:0/3380444923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 msgr2=0x7f076c0064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 --2- 172.21.15.32:0/3380444923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 0x7f076c0064d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07680097f0 tx=0x7f076801c500 comp rx=0 tx=0).stop 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 -- 172.21.15.32:0/3380444923 shutdown_connections 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 --2- 172.21.15.32:0/3380444923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c033b70 0x7f076c0064d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 -- 172.21.15.32:0/3380444923 >> 172.21.15.32:0/3380444923 conn(0x7f076c04d4c0 msgr2=0x7f076c04d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 -- 172.21.15.32:0/3380444923 shutdown_connections 2024-03-16T19:57:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.580+0000 7f077359e640 1 -- 172.21.15.32:0/3380444923 wait complete. 2024-03-16T19:57:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 Processor -- start 2024-03-16T19:57:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 -- start start 2024-03-16T19:57:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 0x7f076c050c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076c035500 0x7f076c04a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c0d5cc0 0x7f076c0dd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c0dcd50 con 0x7f076c035500 2024-03-16T19:57:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c0de640 con 0x7f076c033b70 2024-03-16T19:57:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f077359e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c0df1a0 con 0x7f076c0d5cc0 2024-03-16T19:57:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.584+0000 7f3a61bf6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a5c158800 0x7f3a5c160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a5985b640 1 -- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 msgr2=0x7f3a54158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a5985b640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a540c68e0 0x7f3a54158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a5985b640 1 -- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a540c87b0 msgr2=0x7f3a5414a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a5985b640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a540c87b0 0x7f3a5414a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a5985b640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a44009420 con 0x7f3a5414caa0 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a61bf6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a5c158800 0x7f3a5c160a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59554/0 (socket says 172.21.15.32:59554) 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a61bf6640 1 -- 172.21.15.32:0/232596413 learned_addr learned my addr 172.21.15.32:0/232596413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.588+0000 7f3a613f5640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c0cc5d0 0x7f3a5c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.592+0000 7f945ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 0x7f94601589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59550/0 (socket says 172.21.15.32:59550) 2024-03-16T19:57:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.592+0000 7f945ffff640 1 -- 172.21.15.32:0/4155189746 learned_addr learned my addr 172.21.15.32:0/4155189746 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.596+0000 7f945effd640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9460163d50 0x7f9460204490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.596+0000 7f3a5985b640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5414caa0 0x7f3a5414f2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0042f0 tx=0x7f3a4c0043e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.600+0000 7f3a613f5640 1 -- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a5c158420 msgr2=0x7f3a5c160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.600+0000 7f3a613f5640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a5c158420 0x7f3a5c160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.600+0000 7f3a613f5640 1 -- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a5c158800 msgr2=0x7f3a5c160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.600+0000 7f3a613f5640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a5c158800 0x7f3a5c160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.600+0000 7f3a613f5640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54009b30 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.604+0000 7f2e81d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 0x7f2e84151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.604+0000 7f2e81d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 0x7f2e84151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43814/0 (socket says 172.21.15.32:43814) 2024-03-16T19:57:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.604+0000 7f2e81d9b640 1 -- 172.21.15.32:0/1864344803 learned_addr learned my addr 172.21.15.32:0/1864344803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.608+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f16e401a400 con 0x7f16f0166aa0 2024-03-16T19:57:22.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.608+0000 7f16d7fff640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f16dc066e40 0x7f16dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.608+0000 7f16eeffd640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f16dc066e40 0x7f16dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.608+0000 7f16eeffd640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f16dc066e40 0x7f16dc069300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f16e00b2000 tx=0x7f16e0004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.624+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85ac0b8070 con 0x7f85b80c4090 2024-03-16T19:57:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.624+0000 7f859f7fe640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a4066ec0 0x7f85a4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.624+0000 7f85b67fc640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a4066ec0 0x7f85a4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.624+0000 7f85b67fc640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a4066ec0 0x7f85a4069380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000f70 tx=0x7f85a8007a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f85ac0a9c30 con 0x7f85b80c4090 2024-03-16T19:57:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f859f7fe640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f85a4070530 0x7f85a4072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85a4073040 con 0x7f85a4070530 2024-03-16T19:57:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f3a613f5640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c0cc5d0 0x7f3a5c15c2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b410 tx=0x7f3a5800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f85b77fe640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f85a4070530 0x7f85a4072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f85b77fe640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f85a4070530 0x7f85a4072970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.632+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== osd.1 v2:172.21.15.32:6802/1994708459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f85a4073040 con 0x7f85a4070530 2024-03-16T19:57:22.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f2e81d9b640 1 -- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e84072bd0 msgr2=0x7f2e8414dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f2e81d9b640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e84072bd0 0x7f2e8414dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f2e81d9b640 1 -- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 msgr2=0x7f2e84151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f2e81d9b640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8415a1d0 0x7f2e84151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f2e81d9b640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780094d0 con 0x7f2e84151020 2024-03-16T19:57:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.640+0000 7f945f7fe640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94600c73b0 0x7f9460207a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c00c870 con 0x7f3a5414caa0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c00c9d0 con 0x7f3a5414caa0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a541ebd20 con 0x7f3a5414caa0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c00cd80 con 0x7f3a5414caa0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a541ec2e0 con 0x7f3a5414caa0 2024-03-16T19:57:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a540c70f0 con 0x7f3a5414caa0 2024-03-16T19:57:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.644+0000 7f2e81d9b640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 0x7f2e84151810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2e78019060 tx=0x7f2e7801ac90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f945ffff640 1 -- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9460163d50 msgr2=0x7f9460204490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f945ffff640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9460163d50 0x7f9460204490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f945ffff640 1 -- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94600c73b0 msgr2=0x7f9460207a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f945ffff640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94600c73b0 0x7f9460207a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f945ffff640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94540094d0 con 0x7f9460166aa0 2024-03-16T19:57:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f16e4056470 con 0x7f16f0166aa0 2024-03-16T19:57:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16d7fff640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f16dc070440 0x7f16dc072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16dc072f50 con 0x7f16dc070440 2024-03-16T19:57:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e4009190 con 0x7f16f0166aa0 2024-03-16T19:57:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 --> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85b815a080 con 0x7f85a4070530 2024-03-16T19:57:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16ee7fc640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f16dc070440 0x7f16dc072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f859f7fe640 1 -- 172.21.15.32:0/640350499 <== osd.1 v2:172.21.15.32:6802/1994708459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85b0003f60 con 0x7f85a4070530 2024-03-16T19:57:22.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f16ee7fc640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f16dc070440 0x7f16dc072880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f85a4070530 msgr2=0x7f85a4072970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f85a4070530 0x7f85a4072970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a4066ec0 msgr2=0x7f85a4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a4066ec0 0x7f85a4069380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000f70 tx=0x7f85a8007a90 comp rx=0 tx=0).stop 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 msgr2=0x7f85b806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 --2- 172.21.15.32:0/640350499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85b80c4090 0x7f85b806ea50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85ac01ca90 tx=0x7f85ac01cc20 comp rx=0 tx=0).stop 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b77fe640 1 -- 172.21.15.32:0/640350499 reap_dead start 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 shutdown_connections 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 >> 172.21.15.32:0/640350499 conn(0x7f85b806b190 msgr2=0x7f85b814e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 shutdown_connections 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.648+0000 7f85b7fff640 1 -- 172.21.15.32:0/640350499 wait complete. 2024-03-16T19:57:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.652+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== osd.4 v2:172.21.15.59:6824/446247080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f16dc072f50 con 0x7f16dc070440 2024-03-16T19:57:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.652+0000 7f0771d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076c035500 0x7f076c04a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.656+0000 7f945ffff640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 0x7f94601589b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f945800c7b0 tx=0x7f945800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.656+0000 7f077259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 0x7f076c050c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.656+0000 7f077259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 0x7f076c050c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43684/0 (socket says 172.21.15.32:43684) 2024-03-16T19:57:22.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.656+0000 7f077259c640 1 -- 172.21.15.32:0/2848480953 learned_addr learned my addr 172.21.15.32:0/2848480953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.664+0000 7f077259c640 1 -- 172.21.15.32:0/2848480953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c0d5cc0 msgr2=0x7f076c0dd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.664+0000 7f077259c640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f076c0d5cc0 0x7f076c0dd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.664+0000 7f077259c640 1 -- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076c035500 msgr2=0x7f076c04a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.664+0000 7f077259c640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076c035500 0x7f076c04a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.664+0000 7f077259c640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768009420 con 0x7f076c033b70 2024-03-16T19:57:22.640 INFO:teuthology.orchestra.run.smithi032.stdout:60129542166 2024-03-16T19:57:22.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:57:22.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f077259c640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 0x7f076c050c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f076400c400 tx=0x7f076400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 --> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16d0002d40 con 0x7f16dc070440 2024-03-16T19:57:22.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16d7fff640 1 -- 172.21.15.32:0/2210966094 <== osd.4 v2:172.21.15.59:6824/446247080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f16d0002d40 con 0x7f16dc070440 2024-03-16T19:57:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f16dc070440 msgr2=0x7f16dc072880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f16dc070440 0x7f16dc072880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f16dc066e40 msgr2=0x7f16dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f16dc066e40 0x7f16dc069300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f16e00b2000 tx=0x7f16e0004740 comp rx=0 tx=0).stop 2024-03-16T19:57:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 msgr2=0x7f16f015d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16f5a50640 1 --2- 172.21.15.32:0/2210966094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f0166aa0 0x7f16f015d690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b410 tx=0x7f16e400b8d0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.668+0000 7f16ef7fe640 1 -- 172.21.15.32:0/2210966094 reap_dead start 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.672+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 shutdown_connections 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.672+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 >> 172.21.15.32:0/2210966094 conn(0x7f16f006b190 msgr2=0x7f16f006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.672+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 shutdown_connections 2024-03-16T19:57:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.672+0000 7f16f5a50640 1 -- 172.21.15.32:0/2210966094 wait complete. 2024-03-16T19:57:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58011830 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58011990 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a63659640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c1610c0 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58011d40 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a63659640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c148690 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f3a63659640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a5c16abb0 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764005cf0 con 0x7f076c033b70 2024-03-16T19:57:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764005e50 con 0x7f076c033b70 2024-03-16T19:57:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640072d0 con 0x7f076c033b70 2024-03-16T19:57:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f077359e640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c036b60 con 0x7f076c033b70 2024-03-16T19:57:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.676+0000 7f077359e640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c0debe0 con 0x7f076c033b70 2024-03-16T19:57:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.680+0000 7f077359e640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f076c0344f0 con 0x7f076c033b70 2024-03-16T19:57:22.667 INFO:teuthology.orchestra.run.smithi032.stdout:68719476757 2024-03-16T19:57:22.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:57:22.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff578007450 con 0x7ff58406d1f0 2024-03-16T19:57:22.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7ff56b7fe640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff570066e40 0x7ff570069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7ff582ffd640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff570066e40 0x7ff570069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780b6620 con 0x7f2e84151020 2024-03-16T19:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e780b6780 con 0x7f2e84151020 2024-03-16T19:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780b6b30 con 0x7f2e84151020 2024-03-16T19:57:22.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7ff582ffd640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff570066e40 0x7ff570069300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0070b0 tx=0x7ff57c007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e84073320 con 0x7f2e84151020 2024-03-16T19:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.704+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8414fb50 con 0x7f2e84151020 2024-03-16T19:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.708+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e84154dc0 con 0x7f2e84151020 2024-03-16T19:57:22.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff578055390 con 0x7ff58406d1f0 2024-03-16T19:57:22.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff56b7fe640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7ff570070430 0x7ff570072870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff570072f40 con 0x7ff570070430 2024-03-16T19:57:22.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff5837fe640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7ff570070430 0x7ff570072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff5837fe640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7ff570070430 0x7ff570072870 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.716+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== osd.0 v2:172.21.15.32:6810/1875165640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff570072f40 con 0x7ff570070430 2024-03-16T19:57:22.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.724+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0764007430 con 0x7f076c033b70 2024-03-16T19:57:22.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.724+0000 7f0762ffd640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0754066e70 0x7f0754069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.724+0000 7f0771d9b640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0754066e70 0x7f0754069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.728+0000 7f0771d9b640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0754066e70 0x7f0754069330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07680ad000 tx=0x7f07680a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.728+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0764055690 con 0x7f076c033b70 2024-03-16T19:57:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.728+0000 7f0762ffd640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f0754070460 0x7f07540728a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.728+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0754072f70 con 0x7f0754070460 2024-03-16T19:57:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.728+0000 7f0772d9d640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f0754070460 0x7f07540728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.732+0000 7f0772d9d640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f0754070460 0x7f07540728a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.732+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== osd.2 v2:172.21.15.32:6818/3435819214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0754072f70 con 0x7f0754070460 2024-03-16T19:57:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.740+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3a4c016070 con 0x7f3a5414caa0 2024-03-16T19:57:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.740+0000 7f3a49ffb640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a38066e90 0x7f3a38069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.740+0000 7f3a5905a640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a38066e90 0x7f3a38069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.740+0000 7f3a5905a640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a38066e90 0x7f3a38069350 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a50000f70 tx=0x7f3a500079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.748+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 --> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff584167210 con 0x7ff570070430 2024-03-16T19:57:22.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.748+0000 7ff56b7fe640 1 -- 172.21.15.32:0/524049299 <== osd.0 v2:172.21.15.32:6810/1875165640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff5740af8d0 con 0x7ff570070430 2024-03-16T19:57:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3a4c056d10 con 0x7f3a5414caa0 2024-03-16T19:57:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a49ffb640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f3a38070510 0x7f3a38072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a38073020 con 0x7f3a38070510 2024-03-16T19:57:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7ff570070430 msgr2=0x7ff570072870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7ff570070430 0x7ff570072870 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a58859640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f3a38070510 0x7f3a38072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff570066e40 msgr2=0x7ff570069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff570066e40 0x7ff570069300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0070b0 tx=0x7ff57c007040 comp rx=0 tx=0).stop 2024-03-16T19:57:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 msgr2=0x7ff5840ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 --2- 172.21.15.32:0/524049299 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff58406d1f0 0x7ff5840ca380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c500 tx=0x7ff57800c9c0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a58859640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f3a38070510 0x7f3a38072950 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff5837fe640 1 -- 172.21.15.32:0/524049299 reap_dead start 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 shutdown_connections 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 >> 172.21.15.32:0/524049299 conn(0x7ff58406b190 msgr2=0x7ff5841624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 shutdown_connections 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7ff583fff640 1 -- 172.21.15.32:0/524049299 wait complete. 2024-03-16T19:57:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f077359e640 1 -- 172.21.15.32:0/2848480953 --> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f076c0344f0 con 0x7f0754070460 2024-03-16T19:57:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f0762ffd640 1 -- 172.21.15.32:0/2848480953 <== osd.2 v2:172.21.15.32:6818/3435819214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f076c0344f0 con 0x7f0754070460 2024-03-16T19:57:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== osd.5 v2:172.21.15.59:6816/2401239785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a38073020 con 0x7f3a38070510 2024-03-16T19:57:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f0754070460 msgr2=0x7f07540728a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f0760ff9640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f0754070460 0x7f07540728a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0754066e70 msgr2=0x7f0754069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.760+0000 7f0760ff9640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0754066e70 0x7f0754069330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07680ad000 tx=0x7f07680a6000 comp rx=0 tx=0).stop 2024-03-16T19:57:22.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 msgr2=0x7f076c050c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 --2- 172.21.15.32:0/2848480953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f076c033b70 0x7f076c050c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f076400c400 tx=0x7f076400c8c0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0772d9d640 1 -- 172.21.15.32:0/2848480953 reap_dead start 2024-03-16T19:57:22.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 shutdown_connections 2024-03-16T19:57:22.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 >> 172.21.15.32:0/2848480953 conn(0x7f076c04d4c0 msgr2=0x7f076c0ebd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 shutdown_connections 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.764+0000 7f0760ff9640 1 -- 172.21.15.32:0/2848480953 wait complete. 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.768+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458003d90 con 0x7f9460166aa0 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.768+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9458005070 con 0x7f9460166aa0 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.768+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94600c3830 con 0x7f9460166aa0 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.768+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458005420 con 0x7f9460166aa0 2024-03-16T19:57:22.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.768+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94600c3df0 con 0x7f9460166aa0 2024-03-16T19:57:22.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.784+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f946008bbb0 con 0x7f9460166aa0 2024-03-16T19:57:22.763 INFO:teuthology.orchestra.run.smithi032.stdout:64424509462 2024-03-16T19:57:22.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:57:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:64424509462 2024-03-16T19:57:22.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:57:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.796+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3a5801a400 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.796+0000 7f3a51ffb640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a40066e70 0x7f3a40069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.796+0000 7f3a60bf4640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a40066e70 0x7f3a40069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.796+0000 7f3a60bf4640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a40066e70 0x7f3a40069330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0689e0 tx=0x7f3a4c006c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.808+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9458005580 con 0x7f9460166aa0 2024-03-16T19:57:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.808+0000 7f9447fff640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f944c066e40 0x7f944c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.808+0000 7f945f7fe640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f944c066e40 0x7f944c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.808+0000 7f945f7fe640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f944c066e40 0x7f944c069300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f94500098e0 tx=0x7f9450009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.816+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 --> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a540c70f0 con 0x7f3a38070510 2024-03-16T19:57:22.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.820+0000 7f3a49ffb640 1 -- 172.21.15.32:0/810971291 <== osd.5 v2:172.21.15.59:6816/2401239785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a540c70f0 con 0x7f3a38070510 2024-03-16T19:57:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9458054d70 con 0x7f9460166aa0 2024-03-16T19:57:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f9447fff640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7f944c070440 0x7f944c072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c072f50 con 0x7f944c070440 2024-03-16T19:57:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f945effd640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7f944c070440 0x7f944c072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f3a38070510 msgr2=0x7f3a38072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f3a38070510 0x7f3a38072950 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a38066e90 msgr2=0x7f3a38069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a38066e90 0x7f3a38069350 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a50000f70 tx=0x7f3a500079e0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f945effd640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7f944c070440 0x7f944c072880 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5414caa0 msgr2=0x7f3a5414f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 --2- 172.21.15.32:0/810971291 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5414caa0 0x7f3a5414f2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0042f0 tx=0x7f3a4c0043e0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5985b640 1 -- 172.21.15.32:0/810971291 reap_dead start 2024-03-16T19:57:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 shutdown_connections 2024-03-16T19:57:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.828+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 >> 172.21.15.32:0/810971291 conn(0x7f3a5406afc0 msgr2=0x7f3a540724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.832+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== osd.6 v2:172.21.15.59:6801/459431804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f944c072f50 con 0x7f944c070440 2024-03-16T19:57:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.832+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 shutdown_connections 2024-03-16T19:57:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.832+0000 7f3a5b2be640 1 -- 172.21.15.32:0/810971291 wait complete. 2024-03-16T19:57:22.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.848+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9458005830 con 0x7f9460166aa0 2024-03-16T19:57:22.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.864+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 --> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f946006f2e0 con 0x7f944c070440 2024-03-16T19:57:22.838 INFO:teuthology.orchestra.run.smithi032.stdout:68719476759 2024-03-16T19:57:22.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:57:22.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9447fff640 1 -- 172.21.15.32:0/4155189746 <== osd.6 v2:172.21.15.59:6801/459431804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9454004040 con 0x7f944c070440 2024-03-16T19:57:22.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7f944c070440 msgr2=0x7f944c072880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7f944c070440 0x7f944c072880 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f944c066e40 msgr2=0x7f944c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f944c066e40 0x7f944c069300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f94500098e0 tx=0x7f9450009060 comp rx=0 tx=0).stop 2024-03-16T19:57:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 msgr2=0x7f94601589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 --2- 172.21.15.32:0/4155189746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460166aa0 0x7f94601589b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f945800c7b0 tx=0x7f945800cc70 comp rx=0 tx=0).stop 2024-03-16T19:57:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f945ffff640 1 -- 172.21.15.32:0/4155189746 reap_dead start 2024-03-16T19:57:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f073e73a640 1 Processor -- start 2024-03-16T19:57:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f073e73a640 1 -- start start 2024-03-16T19:57:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 shutdown_connections 2024-03-16T19:57:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 >> 172.21.15.32:0/4155189746 conn(0x7f946006b190 msgr2=0x7f946006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 shutdown_connections 2024-03-16T19:57:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f9465ccc640 1 -- 172.21.15.32:0/4155189746 wait complete. 2024-03-16T19:57:22.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.868+0000 7f073e73a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 0x7f0738071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.872+0000 7f073e73a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0738156d00 con 0x7f0738071950 2024-03-16T19:57:22.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.872+0000 7f073e73a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0738156a20 con 0x7f07380c7810 2024-03-16T19:57:22.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.872+0000 7f073e73a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073806e590 con 0x7f073814fd30 2024-03-16T19:57:22.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.872+0000 7f07377fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0738071950 0x7f073806cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48950/0 (socket says 172.21.15.32:48950) 2024-03-16T19:57:22.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.872+0000 7f07377fe640 1 -- 172.21.15.32:0/432338576 learned_addr learned my addr 172.21.15.32:0/432338576 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 Processor -- start 2024-03-16T19:57:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 -- start start 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda180c6670 0x7fda1806d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda1806d540 0x7fda180c4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda180c3c00 con 0x7fda1814fee0 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda181597e0 con 0x7fda1806d540 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7fda17fff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda18159ac0 con 0x7fda180c6670 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600206490 0 0) 0x7f0738156d00 con 0x7f0738071950 2024-03-16T19:57:22.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.876+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0720003530 con 0x7f0738071950 2024-03-16T19:57:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.880+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401453566 0 0) 0x7f0738156a20 con 0x7f07380c7810 2024-03-16T19:57:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.880+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0738156d00 con 0x7f07380c7810 2024-03-16T19:57:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.880+0000 7fda16ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fda1814fee0 0x7fda181568e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52578/0 (socket says 172.21.15.32:52578) 2024-03-16T19:57:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.880+0000 7fda16ffd640 1 -- 172.21.15.32:0/107515488 learned_addr learned my addr 172.21.15.32:0/107515488 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.880+0000 7fda177fe640 1 --2- 172.21.15.32:0/107515488 >> v2:172.21.15.32:3300/0 conn(0x7fda1806d540 0x7fda180c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.884+0000 7fda177fe640 1 -- 172.21.15.32:0/107515488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda180c6670 msgr2=0x7fda1806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.884+0000 7fda177fe640 1 --2- 172.21.15.32:0/107515488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda180c6670 0x7fda1806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.884+0000 7fda177fe640 1 -- 172.21.15.32:0/107515488 >> v1:172.21.15.59:6789/0 conn(0x7fda1814fee0 legacy=0x7fda181568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.884+0000 7fda177fe640 1 -- 172.21.15.32:0/107515488 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18159da0 con 0x7fda1806d540 2024-03-16T19:57:22.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988991800 0 0) 0x7f0720003530 con 0x7f0738071950 2024-03-16T19:57:22.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 msgr2=0x7f0738071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 0x7f0738071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 >> v1:172.21.15.59:6789/0 conn(0x7f07380c7810 legacy=0x7f0738155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073806e870 con 0x7f0738071950 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f0730005040 con 0x7f0738071950 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 >> v1:172.21.15.32:6789/0 conn(0x7f0738071950 legacy=0x7f073806cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0720003bb0 0x7f0720003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07200047d0 0x7f0720004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 0x7f0720005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073806e590 con 0x7f0720003bb0 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200055d0 con 0x7f07200047d0 2024-03-16T19:57:22.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.888+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0720005ea0 con 0x7f073814fd30 2024-03-16T19:57:22.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.892+0000 7f0737fff640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0720003bb0 0x7f0720003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.892+0000 7f07377fe640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07200047d0 0x7f0720004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7fda177fe640 1 --2- 172.21.15.32:0/107515488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1806d540 0x7fda180c4090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009fd0 tx=0x7fda0c020ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f119b88ddb06ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7f0737fff640 1 -- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 msgr2=0x7f0720005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7f0737fff640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f073814fd30 0x7f0720005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7f0737fff640 1 -- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07200047d0 msgr2=0x7f0720004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7f0737fff640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07200047d0 0x7f0720004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.900+0000 7f0737fff640 1 -- 172.21.15.32:0/432338576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720003530 con 0x7f0720003bb0 2024-03-16T19:57:22.872 INFO:teuthology.orchestra.run.smithi032.stdout:68719476760 2024-03-16T19:57:22.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:57:22.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3a58056390 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a51ffb640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f3a40070460 0x7f3a400728a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a40072f70 con 0x7f3a40070460 2024-03-16T19:57:22.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a61bf6640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f3a40070460 0x7f3a400728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a61bf6640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f3a40070460 0x7f3a400728a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.904+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== osd.3 v2:172.21.15.32:6826/1974622921 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a40072f70 con 0x7f3a40070460 2024-03-16T19:57:22.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7f0737fff640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0720003bb0 0x7f0720003fb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f072c009420 tx=0x7f072c0207d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f24448a4f907a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda15ffb640 1 -- 172.21.15.32:0/107515488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0aaba0 con 0x7fda1806d540 2024-03-16T19:57:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda15ffb640 1 -- 172.21.15.32:0/107515488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c0aad00 con 0x7fda1806d540 2024-03-16T19:57:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda15ffb640 1 -- 172.21.15.32:0/107515488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0b2660 con 0x7fda1806d540 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 -- 172.21.15.32:0/107515488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1806d540 msgr2=0x7fda180c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 --2- 172.21.15.32:0/107515488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1806d540 0x7fda180c4090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009fd0 tx=0x7fda0c020ea0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 -- 172.21.15.32:0/107515488 shutdown_connections 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 --2- 172.21.15.32:0/107515488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1806d540 0x7fda180c4090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 --2- 172.21.15.32:0/107515488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda180c6670 0x7fda1806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 -- 172.21.15.32:0/107515488 >> 172.21.15.32:0/107515488 conn(0x7fda1806b190 msgr2=0x7fda1806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 -- 172.21.15.32:0/107515488 shutdown_connections 2024-03-16T19:57:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.912+0000 7fda17fff640 1 -- 172.21.15.32:0/107515488 wait complete. 2024-03-16T19:57:22.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 Processor -- start 2024-03-16T19:57:22.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 -- start start 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda1806d540 0x7fda18154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1814fee0 0x7fda18151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda180b8a20 con 0x7fda1814fee0 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda18158620 con 0x7fda180c6670 2024-03-16T19:57:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda17fff640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda18157720 con 0x7fda1806d540 2024-03-16T19:57:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda167fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda167fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43696/0 (socket says 172.21.15.32:43696) 2024-03-16T19:57:22.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.916+0000 7fda167fc640 1 -- 172.21.15.32:0/371878935 learned_addr learned my addr 172.21.15.32:0/371878935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c0ac610 con 0x7f0720003bb0 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c0ac790 con 0x7f0720003bb0 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f0736ffd640 1 -- 172.21.15.32:0/432338576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c0acb40 con 0x7f0720003bb0 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 -- 172.21.15.32:0/432338576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0720003bb0 msgr2=0x7f0720003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 --2- 172.21.15.32:0/432338576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0720003bb0 0x7f0720003fb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f072c009420 tx=0x7f072c0207d0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073ccd7640 1 -- 172.21.15.32:0/432338576 reap_dead start 2024-03-16T19:57:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7fda177fe640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1814fee0 0x7fda18151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 -- 172.21.15.32:0/432338576 shutdown_connections 2024-03-16T19:57:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 -- 172.21.15.32:0/432338576 >> 172.21.15.32:0/432338576 conn(0x7f073806b190 msgr2=0x7f073806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 -- 172.21.15.32:0/432338576 shutdown_connections 2024-03-16T19:57:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.920+0000 7f073e73a640 1 -- 172.21.15.32:0/432338576 wait complete. 2024-03-16T19:57:22.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 Processor -- start 2024-03-16T19:57:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 -- start start 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0720003bb0 0x7f073814c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0738159da0 0x7f0738144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07381ef000 con 0x7f0738159da0 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07381ee6c0 con 0x7f0738158f50 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7f073e73a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0738159500 con 0x7f0720003bb0 2024-03-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7fda167fc640 1 -- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda1806d540 msgr2=0x7fda18154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7fda167fc640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda1806d540 0x7fda18154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7fda167fc640 1 -- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1814fee0 msgr2=0x7fda18151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7fda167fc640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1814fee0 0x7fda18151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.924+0000 7fda167fc640 1 -- 172.21.15.32:0/371878935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c009cb0 con 0x7fda180c6670 2024-03-16T19:57:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.928+0000 7f07377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.928+0000 7fda167fc640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda0800b550 tx=0x7fda0800ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.928+0000 7f07377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43704/0 (socket says 172.21.15.32:43704) 2024-03-16T19:57:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.928+0000 7f07377fe640 1 -- 172.21.15.32:0/3134829285 learned_addr learned my addr 172.21.15.32:0/3134829285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.932+0000 7f073ccd7640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0738159da0 0x7f0738144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a58011ea0 con 0x7f3a5c0cc5d0 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f07377fe640 1 -- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0720003bb0 msgr2=0x7f073814c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f07377fe640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0720003bb0 0x7f073814c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f07377fe640 1 -- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0738159da0 msgr2=0x7f0738144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f07377fe640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0738159da0 0x7f0738144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.936+0000 7f07377fe640 1 -- 172.21.15.32:0/3134829285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07381597e0 con 0x7f0738158f50 2024-03-16T19:57:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.940+0000 7f3a63659640 1 -- 172.21.15.32:0/232596413 --> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a5c149610 con 0x7f3a40070460 2024-03-16T19:57:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.940+0000 7f3a51ffb640 1 -- 172.21.15.32:0/232596413 <== osd.3 v2:172.21.15.32:6826/1974622921 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a5c149610 con 0x7f3a40070460 2024-03-16T19:57:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08010870 con 0x7fda180c6670 2024-03-16T19:57:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda18157160 con 0x7fda180c6670 2024-03-16T19:57:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7f07377fe640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0730001340 tx=0x7f0730005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda18151210 con 0x7fda180c6670 2024-03-16T19:57:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda080109d0 con 0x7fda180c6670 2024-03-16T19:57:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.944+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08010d40 con 0x7fda180c6670 2024-03-16T19:57:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.952+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f80052a0 con 0x7fda180c6670 2024-03-16T19:57:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f3a40070460 msgr2=0x7f3a400728a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f3a40070460 0x7f3a400728a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a40066e70 msgr2=0x7f3a40069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a40066e70 0x7f3a40069330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0689e0 tx=0x7f3a4c006c00 comp rx=0 tx=0).stop 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c0cc5d0 msgr2=0x7f3a5c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 --2- 172.21.15.32:0/232596413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a5c0cc5d0 0x7f3a5c15c2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b410 tx=0x7f3a5800b8d0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a61bf6640 1 -- 172.21.15.32:0/232596413 reap_dead start 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 shutdown_connections 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 >> 172.21.15.32:0/232596413 conn(0x7f3a5c066ea0 msgr2=0x7f3a5c0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 shutdown_connections 2024-03-16T19:57:22.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.956+0000 7f3a477fe640 1 -- 172.21.15.32:0/232596413 wait complete. 2024-03-16T19:57:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fda08019450 con 0x7fda180c6670 2024-03-16T19:57:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7fd9ff7fe640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 0x7fda0006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2e780b2040 con 0x7f2e84151020 2024-03-16T19:57:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7f2e72ffd640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e64066e70 0x7f2e64069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7f2e8259c640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e64066e70 0x7f2e64069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7fda16ffd640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 0x7fda0006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7f2e8259c640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e64066e70 0x7f2e64069330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e74007fd0 tx=0x7f2e740079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.980+0000 7fda16ffd640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 0x7fda0006d860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda10001fe0 tx=0x7fda100015d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.954 INFO:teuthology.orchestra.run.smithi032.stdout:64424509460 2024-03-16T19:57:22.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-16T19:57:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fda08024e30 con 0x7fda180c6670 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2e780f0630 con 0x7f2e84151020 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f2e72ffd640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7f2e64070460 0x7f2e640728a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e64072f70 con 0x7f2e64070460 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730004f80 con 0x7f0738158f50 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730001d50 con 0x7f0738158f50 2024-03-16T19:57:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07300020e0 con 0x7f0738158f50 2024-03-16T19:57:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07381ee940 con 0x7f0738158f50 2024-03-16T19:57:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073814cee0 con 0x7f0738158f50 2024-03-16T19:57:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.984+0000 7f2e82d9d640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7f2e64070460 0x7f2e640728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.988+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e780ec1d0 con 0x7f2e84151020 2024-03-16T19:57:22.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.988+0000 7f2e82d9d640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7f2e64070460 0x7f2e640728a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:22.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.988+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== osd.7 v2:172.21.15.59:6800/1528567805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e64072f70 con 0x7f2e64070460 2024-03-16T19:57:22.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.988+0000 7f0680a14640 1 Processor -- start 2024-03-16T19:57:22.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.988+0000 7f0680a14640 1 -- start start 2024-03-16T19:57:22.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f0680a14640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f067c158340 0x7f067c0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f0680a14640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f067c0c73b0 0x7f067c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f0680a14640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c15f070 con 0x7f067c06cbb0 2024-03-16T19:57:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f0680a14640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c15e460 con 0x7f067c158340 2024-03-16T19:57:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f0680a14640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c15e740 con 0x7f067c0c73b0 2024-03-16T19:57:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:22.992+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073804f2d0 con 0x7f0738158f50 2024-03-16T19:57:22.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.012+0000 7ff32cf38640 1 Processor -- start 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7f067ad9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f067c06cbb0 0x7f067c06e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48960/0 (socket says 172.21.15.32:48960) 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7f067ad9d640 1 -- 172.21.15.32:0/1931832591 learned_addr learned my addr 172.21.15.32:0/1931832591 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 -- start start 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff32806d640 0x7ff3280c6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff32814f9d0 0x7ff32814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff328159500 con 0x7ff32806d260 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff328150360 con 0x7ff32814f9d0 2024-03-16T19:57:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7ff32cf38640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3281597e0 con 0x7ff32806d640 2024-03-16T19:57:22.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.016+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 --> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e840731a0 con 0x7f2e64070460 2024-03-16T19:57:22.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e72ffd640 1 -- 172.21.15.32:0/1864344803 <== osd.7 v2:172.21.15.59:6800/1528567805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e7c006b20 con 0x7f2e64070460 2024-03-16T19:57:22.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917673631 0 0) 0x7f067c15f070 con 0x7f067c06cbb0 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664003530 con 0x7f067c06cbb0 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7f2e64070460 msgr2=0x7f2e640728a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7f2e64070460 0x7f2e640728a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e64066e70 msgr2=0x7f2e64069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e64066e70 0x7f2e64069330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e74007fd0 tx=0x7f2e740079a0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 msgr2=0x7f2e84151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:22.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 --2- 172.21.15.32:0/1864344803 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e84151020 0x7f2e84151810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2e78019060 tx=0x7f2e7801ac90 comp rx=0 tx=0).stop 2024-03-16T19:57:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e82d9d640 1 -- 172.21.15.32:0/1864344803 reap_dead start 2024-03-16T19:57:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 shutdown_connections 2024-03-16T19:57:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 >> 172.21.15.32:0/1864344803 conn(0x7f2e840cc7a0 msgr2=0x7f2e840c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 shutdown_connections 2024-03-16T19:57:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.020+0000 7f2e88e0d640 1 -- 172.21.15.32:0/1864344803 wait complete. 2024-03-16T19:57:22.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.024+0000 7ff32659c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff32806d260 0x7ff328156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48966/0 (socket says 172.21.15.32:48966) 2024-03-16T19:57:22.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.024+0000 7ff32659c640 1 -- 172.21.15.32:0/4260514362 learned_addr learned my addr 172.21.15.32:0/4260514362 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067a59c640 1 --2- 172.21.15.32:0/1931832591 >> v2:172.21.15.32:3300/0 conn(0x7f067c158340 0x7f067c0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384083037 0 0) 0x7f0664003530 con 0x7f067c06cbb0 2024-03-16T19:57:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 >> v2:172.21.15.59:3300/0 conn(0x7f067c0c73b0 msgr2=0x7f067c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 --2- 172.21.15.32:0/1931832591 >> v2:172.21.15.59:3300/0 conn(0x7f067c0c73b0 0x7f067c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 >> v2:172.21.15.32:3300/0 conn(0x7f067c158340 msgr2=0x7f067c0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 --2- 172.21.15.32:0/1931832591 >> v2:172.21.15.32:3300/0 conn(0x7f067c158340 0x7f067c0c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c15ea20 con 0x7f067c06cbb0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f0674001a00 con 0x7f067c06cbb0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 >> v1:172.21.15.32:6789/0 conn(0x7f067c06cbb0 legacy=0x7f067c06e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06640039f0 0x7f0664003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 0x7f0664004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640052e0 0x7f066400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15e460 con 0x7f06640052e0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15e740 con 0x7f06640039f0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.028+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066400c2a0 con 0x7f06640047d0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 0x7f0664004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.036+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321632426 0 0) 0x7ff328159500 con 0x7ff32806d260 2024-03-16T19:57:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.036+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff310003530 con 0x7ff32806d260 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 -- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06640039f0 msgr2=0x7f0664003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06640039f0 0x7f0664003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 -- 172.21.15.32:0/1931832591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640052e0 msgr2=0x7f066400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640052e0 0x7f066400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.032+0000 7f0679d9b640 1 -- 172.21.15.32:0/1931832591 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664003530 con 0x7f06640047d0 2024-03-16T19:57:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.036+0000 7f067ad9d640 1 -- 172.21.15.32:0/1931832591 reap_dead start 2024-03-16T19:57:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7f0679d9b640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 0x7f0664004c50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f066c009890 tx=0x7f066c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2247e7bfb460a251 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498696181 0 0) 0x7ff310003530 con 0x7ff32806d260 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 >> v2:172.21.15.59:3300/0 conn(0x7ff32806d640 msgr2=0x7ff3280c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 --2- 172.21.15.32:0/4260514362 >> v2:172.21.15.59:3300/0 conn(0x7ff32806d640 0x7ff3280c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 >> v2:172.21.15.32:3300/0 conn(0x7ff32814f9d0 msgr2=0x7ff32814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 --2- 172.21.15.32:0/4260514362 >> v2:172.21.15.32:3300/0 conn(0x7ff32814f9d0 0x7ff32814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328159ac0 con 0x7ff32806d260 2024-03-16T19:57:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7ff318001680 con 0x7ff32806d260 2024-03-16T19:57:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 >> v1:172.21.15.32:6789/0 conn(0x7ff32806d260 legacy=0x7ff328156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff310003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100047d0 0x7ff310004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3100052e0 0x7ff310007970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff328150360 con 0x7ff3100052e0 2024-03-16T19:57:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3281597e0 con 0x7ff3100047d0 2024-03-16T19:57:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.040+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff310008280 con 0x7ff3100039f0 2024-03-16T19:57:23.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c004230 con 0x7f06640047d0 2024-03-16T19:57:23.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0a7050 con 0x7f06640047d0 2024-03-16T19:57:23.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7f067959a640 1 -- 172.21.15.32:0/1931832591 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c003600 con 0x7f06640047d0 2024-03-16T19:57:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7ff32659c640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff310003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7f0680a14640 1 -- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 msgr2=0x7f0664004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.044+0000 7f0680a14640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 0x7f0664004c50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f066c009890 tx=0x7f066c01c3e0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 -- 172.21.15.32:0/1931832591 shutdown_connections 2024-03-16T19:57:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 --2- 172.21.15.32:0/1931832591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06640047d0 0x7f0664004c50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 -- 172.21.15.32:0/1931832591 >> 172.21.15.32:0/1931832591 conn(0x7f067c06b190 msgr2=0x7f067c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 -- 172.21.15.32:0/1931832591 shutdown_connections 2024-03-16T19:57:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 -- 172.21.15.32:0/1931832591 wait complete. 2024-03-16T19:57:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 Processor -- start 2024-03-16T19:57:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 -- start start 2024-03-16T19:57:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.048+0000 7f0680a14640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 0x7f067c13fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0680a14640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 0x7f067c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0680a14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f067c15b0e0 0x7f067c139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0680a14640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c140930 con 0x7f06640047d0 2024-03-16T19:57:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0680a14640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15b590 con 0x7f067c15b0e0 2024-03-16T19:57:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0680a14640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15b750 con 0x7f067c06f6f0 2024-03-16T19:57:23.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7f0679d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 0x7f067c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.056+0000 7f0679d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 0x7f067c13dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43892/0 (socket says 172.21.15.32:43892) 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.056+0000 7f0679d9b640 1 -- 172.21.15.32:0/177392639 learned_addr learned my addr 172.21.15.32:0/177392639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff32659c640 1 -- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100047d0 msgr2=0x7ff310004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff32659c640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100047d0 0x7ff310004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff32659c640 1 -- 172.21.15.32:0/4260514362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3100052e0 msgr2=0x7ff310007970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff32659c640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3100052e0 0x7ff310007970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff32659c640 1 -- 172.21.15.32:0/4260514362 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff310003530 con 0x7ff3100039f0 2024-03-16T19:57:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.052+0000 7ff326d9d640 1 -- 172.21.15.32:0/4260514362 reap_dead start 2024-03-16T19:57:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.056+0000 7f067a59c640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 0x7f067c13fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.027 INFO:teuthology.orchestra.run.smithi032.stdout:68719476758 2024-03-16T19:57:23.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-16T19:57:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 -- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 msgr2=0x7f067c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 0x7f067c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 -- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f067c15b0e0 msgr2=0x7f067c139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f067c15b0e0 0x7f067c139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 -- 172.21.15.32:0/177392639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c13eff0 con 0x7f06640047d0 2024-03-16T19:57:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.060+0000 7f067a59c640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 0x7f067c13fc60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f067000c5a0 tx=0x7f067000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7ff32659c640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff310003e10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3180050c0 tx=0x7ff31801d2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6dd9450def56d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670004230 con 0x7f06640047d0 2024-03-16T19:57:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670004390 con 0x7f06640047d0 2024-03-16T19:57:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670003860 con 0x7f06640047d0 2024-03-16T19:57:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c143d90 con 0x7f06640047d0 2024-03-16T19:57:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c13a6c0 con 0x7f06640047d0 2024-03-16T19:57:23.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.068+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600052a0 con 0x7f06640047d0 2024-03-16T19:57:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180ad620 con 0x7ff3100039f0 2024-03-16T19:57:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3180a9070 con 0x7ff3100039f0 2024-03-16T19:57:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32559a640 1 -- 172.21.15.32:0/4260514362 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180ada20 con 0x7ff3100039f0 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 -- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 msgr2=0x7ff310003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff310003e10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3180050c0 tx=0x7ff31801d2b0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 -- 172.21.15.32:0/4260514362 shutdown_connections 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 --2- 172.21.15.32:0/4260514362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff310003e10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 -- 172.21.15.32:0/4260514362 >> 172.21.15.32:0/4260514362 conn(0x7ff32806b190 msgr2=0x7ff32806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 -- 172.21.15.32:0/4260514362 shutdown_connections 2024-03-16T19:57:23.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.072+0000 7ff32cf38640 1 -- 172.21.15.32:0/4260514362 wait complete. 2024-03-16T19:57:23.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 Processor -- start 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 -- start start 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff328141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3280c9a50 0x7ff328070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 0x7ff328071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3281536c0 con 0x7ff3280c9e90 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3281533e0 con 0x7ff3280c9a50 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.076+0000 7ff32cf38640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff328141e80 con 0x7ff3100039f0 2024-03-16T19:57:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.080+0000 7ff32659c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff328141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.080+0000 7ff32659c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff328141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43904/0 (socket says 172.21.15.32:43904) 2024-03-16T19:57:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.080+0000 7ff32659c640 1 -- 172.21.15.32:0/1941148891 learned_addr learned my addr 172.21.15.32:0/1941148891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.080+0000 7ff326d9d640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 0x7ff328071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.084+0000 7ff326d9d640 1 -- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 msgr2=0x7ff328141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.084+0000 7ff326d9d640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff328141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.084+0000 7ff326d9d640 1 -- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3280c9a50 msgr2=0x7ff328070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.084+0000 7ff326d9d640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3280c9a50 0x7ff328070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.084+0000 7ff326d9d640 1 -- 172.21.15.32:0/1941148891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328142160 con 0x7ff3280c9e90 2024-03-16T19:57:23.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 Processor -- start 2024-03-16T19:57:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 -- start start 2024-03-16T19:57:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0a6015e950 0x7f0a60160d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a60156a60 con 0x7f0a600c68e0 2024-03-16T19:57:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6014ce80 con 0x7f0a6015e950 2024-03-16T19:57:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7f0a68570640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60156c90 con 0x7f0a6014caa0 2024-03-16T19:57:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.088+0000 7ff326d9d640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 0x7ff328071520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff3280c3330 tx=0x7ff32000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.092+0000 7f0a6630c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.092+0000 7f0a6630c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43912/0 (socket says 172.21.15.32:43912) 2024-03-16T19:57:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.092+0000 7f0a6630c640 1 -- 172.21.15.32:0/2398630997 learned_addr learned my addr 172.21.15.32:0/2398630997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.096+0000 7f0a6630c640 1 -- 172.21.15.32:0/2398630997 >> v2:172.21.15.59:3300/0 conn(0x7f0a6015e950 msgr2=0x7f0a60160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.096+0000 7f0a6630c640 1 --2- 172.21.15.32:0/2398630997 >> v2:172.21.15.59:3300/0 conn(0x7f0a6015e950 0x7f0a60160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.096+0000 7f0a6630c640 1 -- 172.21.15.32:0/2398630997 >> v1:172.21.15.59:6789/0 conn(0x7f0a600c68e0 legacy=0x7f0a600c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.096+0000 7f0a6630c640 1 -- 172.21.15.32:0/2398630997 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60156f30 con 0x7f0a6014caa0 2024-03-16T19:57:23.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.104+0000 7f0a6630c640 1 --2- 172.21.15.32:0/2398630997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a50008870 tx=0x7f0a50021240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ac7dfd7e58f0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a6530a640 1 -- 172.21.15.32:0/2398630997 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500b3410 con 0x7f0a6014caa0 2024-03-16T19:57:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a6530a640 1 -- 172.21.15.32:0/2398630997 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500ad030 con 0x7f0a6014caa0 2024-03-16T19:57:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a6530a640 1 -- 172.21.15.32:0/2398630997 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500b3790 con 0x7f0a6014caa0 2024-03-16T19:57:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- 172.21.15.32:0/2398630997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 msgr2=0x7f0a60159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 --2- 172.21.15.32:0/2398630997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a50008870 tx=0x7f0a50021240 comp rx=0 tx=0).stop 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- 172.21.15.32:0/2398630997 shutdown_connections 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 --2- 172.21.15.32:0/2398630997 >> v2:172.21.15.59:3300/0 conn(0x7f0a6015e950 0x7f0a60160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 --2- 172.21.15.32:0/2398630997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a6014caa0 0x7f0a60159e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- 172.21.15.32:0/2398630997 >> 172.21.15.32:0/2398630997 conn(0x7f0a6006afc0 msgr2=0x7f0a6006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- 172.21.15.32:0/2398630997 shutdown_connections 2024-03-16T19:57:23.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- 172.21.15.32:0/2398630997 wait complete. 2024-03-16T19:57:23.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 Processor -- start 2024-03-16T19:57:23.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.112+0000 7f0a68570640 1 -- start start 2024-03-16T19:57:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6014caa0 0x7f0a601571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015e950 0x7f0a601edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a601581f0 con 0x7f0a6014caa0 2024-03-16T19:57:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60157be0 con 0x7f0a6015e950 2024-03-16T19:57:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a68570640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60158cf0 con 0x7f0a600c68e0 2024-03-16T19:57:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a6630c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a6630c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43924/0 (socket says 172.21.15.32:43924) 2024-03-16T19:57:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.116+0000 7f0a6630c640 1 -- 172.21.15.32:0/7559647 learned_addr learned my addr 172.21.15.32:0/7559647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 Processor -- start 2024-03-16T19:57:23.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 -- start start 2024-03-16T19:57:23.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba58150340 0x7fba581560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba580c75d0 con 0x7fba580c77f0 2024-03-16T19:57:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba580c6c70 con 0x7fba58150340 2024-03-16T19:57:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.120+0000 7fba6051f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5815a270 con 0x7fba580c4260 2024-03-16T19:57:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.124+0000 7f0a6630c640 1 -- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015e950 msgr2=0x7f0a601edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.124+0000 7f0a6630c640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015e950 0x7f0a601edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.124+0000 7f0a6630c640 1 -- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6014caa0 msgr2=0x7f0a601571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.124+0000 7f0a6630c640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6014caa0 0x7f0a601571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.124+0000 7f0a6630c640 1 -- 172.21.15.32:0/7559647 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50008550 con 0x7f0a600c68e0 2024-03-16T19:57:23.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.128+0000 7f0a6630c640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a50008870 tx=0x7f0a5001fa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.136+0000 7fba5daba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.136+0000 7fba5daba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43928/0 (socket says 172.21.15.32:43928) 2024-03-16T19:57:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.136+0000 7fba5daba640 1 -- 172.21.15.32:0/3099243333 learned_addr learned my addr 172.21.15.32:0/3099243333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500c2070 con 0x7f0a600c68e0 2024-03-16T19:57:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500b3c00 con 0x7f0a600c68e0 2024-03-16T19:57:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a60158fd0 con 0x7f0a600c68e0 2024-03-16T19:57:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5000b0b0 con 0x7f0a600c68e0 2024-03-16T19:57:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a60157e80 con 0x7f0a600c68e0 2024-03-16T19:57:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.140+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60074770 con 0x7f0a600c68e0 2024-03-16T19:57:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.144+0000 7fba5daba640 1 -- 172.21.15.32:0/3099243333 >> v2:172.21.15.32:3300/0 conn(0x7fba58150340 msgr2=0x7fba581560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.144+0000 7fba5daba640 1 --2- 172.21.15.32:0/3099243333 >> v2:172.21.15.32:3300/0 conn(0x7fba58150340 0x7fba581560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.144+0000 7fba5daba640 1 -- 172.21.15.32:0/3099243333 >> v1:172.21.15.32:6789/0 conn(0x7fba580c77f0 legacy=0x7fba581596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.144+0000 7fba5daba640 1 -- 172.21.15.32:0/3099243333 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5815a550 con 0x7fba580c4260 2024-03-16T19:57:23.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.164+0000 7fdeb17a2640 1 Processor -- start 2024-03-16T19:57:23.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.164+0000 7fdeb17a2640 1 -- start start 2024-03-16T19:57:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.164+0000 7fdeb17a2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.164+0000 7fdeb17a2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdeac070430 0x7fdeac06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.168+0000 7fdeb17a2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac15abe0 con 0x7fdeac06c840 2024-03-16T19:57:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.168+0000 7fdeb17a2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac070b90 con 0x7fdeac070430 2024-03-16T19:57:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.168+0000 7fdeb17a2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac159fd0 con 0x7fdeac1540e0 2024-03-16T19:57:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.172+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f06700039c0 con 0x7f06640047d0 2024-03-16T19:57:23.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.172+0000 7f066affd640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 0x7f065c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.172+0000 7f0679d9b640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 0x7f065c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.172+0000 7f0679d9b640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 0x7f065c069380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f067c06faf0 tx=0x7f066c004850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.184+0000 7fba5daba640 1 --2- 172.21.15.32:0/3099243333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba48007360 tx=0x7fba4801fd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae1b1928b1b7befc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.184+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0a5000b210 con 0x7f0a600c68e0 2024-03-16T19:57:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.184+0000 7f0a56ffd640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 0x7f0a44069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.184+0000 7f0a65b0b640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 0x7f0a44069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.184+0000 7f0a65b0b640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 0x7f0a44069330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a6014a110 tx=0x7f0a5c006bb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.212+0000 7f3f863b3640 1 Processor -- start 2024-03-16T19:57:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.212+0000 7f3f863b3640 1 -- start start 2024-03-16T19:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.216+0000 7f3f863b3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3f80150f00 0x7f3f800c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.216+0000 7f3f863b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f800c3540 0x7f3f800c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.216+0000 7f3f863b3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f800c7e50 con 0x7f3f800c7040 2024-03-16T19:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.216+0000 7f3f863b3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f80070850 con 0x7f3f80150f00 2024-03-16T19:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.216+0000 7f3f863b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801512e0 con 0x7f3f800c3540 2024-03-16T19:57:23.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.324+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fda080559b0 con 0x7fda180c6670 2024-03-16T19:57:23.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.396+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0670055850 con 0x7f06640047d0 2024-03-16T19:57:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32000a810 con 0x7ff3280c9e90 2024-03-16T19:57:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32000a990 con 0x7ff3280c9e90 2024-03-16T19:57:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32813f1d0 con 0x7ff3280c9e90 2024-03-16T19:57:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32000ad40 con 0x7ff3280c9e90 2024-03-16T19:57:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3281415d0 con 0x7ff3280c9e90 2024-03-16T19:57:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7f3f7ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3f80150f00 0x7f3f800c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7f3f7ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3f80150f00 0x7f3f800c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59712/0 (socket says 172.21.15.32:59712) 2024-03-16T19:57:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7f3f7ffff640 1 -- 172.21.15.32:0/3905568258 learned_addr learned my addr 172.21.15.32:0/3905568258 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.408+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32813f020 con 0x7ff3280c9e90 2024-03-16T19:57:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.420+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0670051af0 con 0x7f06640047d0 2024-03-16T19:57:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.424+0000 7f3f7ffff640 1 -- 172.21.15.32:0/3905568258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f800c3540 msgr2=0x7f3f800c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.424+0000 7f3f7ffff640 1 --2- 172.21.15.32:0/3905568258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f800c3540 0x7f3f800c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.424+0000 7f3f7ffff640 1 -- 172.21.15.32:0/3905568258 >> v1:172.21.15.59:6789/0 conn(0x7f3f800c7040 legacy=0x7f3f800c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.424+0000 7f3f7ffff640 1 -- 172.21.15.32:0/3905568258 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80070290 con 0x7f3f80150f00 2024-03-16T19:57:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.468+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0a500f4830 con 0x7f0a600c68e0 2024-03-16T19:57:23.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7fba5d2b9640 1 -- 172.21.15.32:0/3099243333 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48008be0 con 0x7fba580c4260 2024-03-16T19:57:23.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7fba5d2b9640 1 -- 172.21.15.32:0/3099243333 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba48008d40 con 0x7fba580c4260 2024-03-16T19:57:23.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7fba5d2b9640 1 -- 172.21.15.32:0/3099243333 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba480a9b10 con 0x7fba580c4260 2024-03-16T19:57:23.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7fba6051f640 1 -- 172.21.15.32:0/3099243333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 msgr2=0x7fba580c7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7fba6051f640 1 --2- 172.21.15.32:0/3099243333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba48007360 tx=0x7fba4801fd20 comp rx=0 tx=0).stop 2024-03-16T19:57:23.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.488+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0a500f03d0 con 0x7f0a600c68e0 2024-03-16T19:57:23.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fdeaaffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fdeaaffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43930/0 (socket says 172.21.15.32:43930) 2024-03-16T19:57:23.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fdeaaffd640 1 -- 172.21.15.32:0/1906404094 learned_addr learned my addr 172.21.15.32:0/1906404094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 -- 172.21.15.32:0/3099243333 shutdown_connections 2024-03-16T19:57:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 --2- 172.21.15.32:0/3099243333 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c4260 0x7fba580c7090 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 --2- 172.21.15.32:0/3099243333 >> v2:172.21.15.32:3300/0 conn(0x7fba58150340 0x7fba581560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 -- 172.21.15.32:0/3099243333 >> 172.21.15.32:0/3099243333 conn(0x7fba580c83d0 msgr2=0x7fba580c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 -- 172.21.15.32:0/3099243333 shutdown_connections 2024-03-16T19:57:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.496+0000 7fba6051f640 1 -- 172.21.15.32:0/3099243333 wait complete. 2024-03-16T19:57:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fba6051f640 1 Processor -- start 2024-03-16T19:57:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fba6051f640 1 -- start start 2024-03-16T19:57:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fba6051f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba580c4260 0x7fba58150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fdeaaffd640 1 -- 172.21.15.32:0/1906404094 >> v2:172.21.15.59:3300/0 conn(0x7fdeac070430 msgr2=0x7fdeac06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fdeaaffd640 1 --2- 172.21.15.32:0/1906404094 >> v2:172.21.15.59:3300/0 conn(0x7fdeac070430 0x7fdeac06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fdeaaffd640 1 -- 172.21.15.32:0/1906404094 >> v1:172.21.15.59:6789/0 conn(0x7fdeac06c840 legacy=0x7fdeac15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.500+0000 7fdeaaffd640 1 -- 172.21.15.32:0/1906404094 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac15a2b0 con 0x7fdeac1540e0 2024-03-16T19:57:23.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba6051f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba6051f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba58150340 0x7fba581f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba6051f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba581fd4e0 con 0x7fba580c4260 2024-03-16T19:57:23.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba6051f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5814a9d0 con 0x7fba58150340 2024-03-16T19:57:23.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba6051f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba58151470 con 0x7fba580c77f0 2024-03-16T19:57:23.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.504+0000 7fba5daba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.508+0000 7fba5daba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43942/0 (socket says 172.21.15.32:43942) 2024-03-16T19:57:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.508+0000 7fba5daba640 1 -- 172.21.15.32:0/316605833 learned_addr learned my addr 172.21.15.32:0/316605833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.508+0000 7fdeaaffd640 1 --2- 172.21.15.32:0/1906404094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009870 tx=0x7fde9c01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257a3934e09b106a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.516+0000 7fba5daba640 1 -- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba58150340 msgr2=0x7fba581f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.516+0000 7fba5daba640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba58150340 0x7fba581f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.516+0000 7fba5daba640 1 -- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba580c4260 msgr2=0x7fba58150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.516+0000 7fba5daba640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba580c4260 0x7fba58150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.516+0000 7fba5daba640 1 -- 172.21.15.32:0/316605833 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48007040 con 0x7fba580c77f0 2024-03-16T19:57:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeaa7fc640 1 -- 172.21.15.32:0/1906404094 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c0a89d0 con 0x7fdeac1540e0 2024-03-16T19:57:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeaa7fc640 1 -- 172.21.15.32:0/1906404094 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c0a8b30 con 0x7fdeac1540e0 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeaa7fc640 1 -- 172.21.15.32:0/1906404094 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c0b0440 con 0x7fdeac1540e0 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1906404094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 msgr2=0x7fdeac072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1906404094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009870 tx=0x7fde9c01ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1906404094 shutdown_connections 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1906404094 >> v2:172.21.15.59:3300/0 conn(0x7fdeac070430 0x7fdeac06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1906404094 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac072b30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1906404094 >> 172.21.15.32:0/1906404094 conn(0x7fdeac0cc7a0 msgr2=0x7fdeac0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1906404094 shutdown_connections 2024-03-16T19:57:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1906404094 wait complete. 2024-03-16T19:57:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 Processor -- start 2024-03-16T19:57:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fba5daba640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fba4801e3d0 tx=0x7fba4801cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.520+0000 7fdeb17a2640 1 -- start start 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdeac06c840 0x7fdeac1fce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac070430 0x7fdeac15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac1fd3a0 con 0x7fdeac06c840 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac1571a0 con 0x7fdeac070430 2024-03-16T19:57:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.524+0000 7fdeb17a2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac157460 con 0x7fdeac1540e0 2024-03-16T19:57:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fdeab7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba480c0070 con 0x7fba580c77f0 2024-03-16T19:57:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba48007ca0 con 0x7fba580c77f0 2024-03-16T19:57:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba58151730 con 0x7fba580c77f0 2024-03-16T19:57:23.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.532+0000 7fdeab7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43954/0 (socket says 172.21.15.32:43954) 2024-03-16T19:57:23.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.532+0000 7fdeab7fe640 1 -- 172.21.15.32:0/1484318700 learned_addr learned my addr 172.21.15.32:0/1484318700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba581ffad0 con 0x7fba580c77f0 2024-03-16T19:57:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.528+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800a310 con 0x7fba580c77f0 2024-03-16T19:57:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.532+0000 7f3f7ffff640 1 --2- 172.21.15.32:0/3905568258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f80150f00 0x7f3f800c88d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000bb10 tx=0x7f3f7001ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec321b75ea8d1686 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.540+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba400052a0 con 0x7fba580c77f0 2024-03-16T19:57:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.556+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff320005d50 con 0x7ff3280c9e90 2024-03-16T19:57:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.556+0000 7ff316ffd640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 0x7ff308069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.556+0000 7ff32659c640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 0x7ff308069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.556+0000 7ff32659c640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 0x7ff308069350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff31801d0e0 tx=0x7ff318001900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.572+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fba4800a470 con 0x7fba580c77f0 2024-03-16T19:57:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.572+0000 7fba4effd640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 0x7fba3c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.572+0000 7fba5e2bb640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 0x7fba3c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.572+0000 7fba5e2bb640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 0x7fba3c069380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba54009730 tx=0x7fba54006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.576+0000 7fdeab7fe640 1 -- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac070430 msgr2=0x7fdeac15c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.576+0000 7fdeab7fe640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac070430 0x7fdeac15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.576+0000 7fdeab7fe640 1 -- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdeac06c840 msgr2=0x7fdeac1fce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.576+0000 7fdeab7fe640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdeac06c840 0x7fdeac1fce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.576+0000 7fdeab7fe640 1 -- 172.21.15.32:0/1484318700 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c0094d0 con 0x7fdeac1540e0 2024-03-16T19:57:23.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.584+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fba480b9e90 con 0x7fba580c77f0 2024-03-16T19:57:23.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.592+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff320054eb0 con 0x7ff3280c9e90 2024-03-16T19:57:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.616+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0660005820 con 0x7f06640047d0 2024-03-16T19:57:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.620+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f073001e460 con 0x7f0738158f50 2024-03-16T19:57:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.620+0000 7f0734ff9640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 0x7f0724069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.620+0000 7f0737fff640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 0x7f0724069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.620+0000 7f0737fff640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 0x7f0724069380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b3000 tx=0x7f072c0ae040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.624+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0730056d60 con 0x7f0738158f50 2024-03-16T19:57:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.640+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd9f8005820 con 0x7fda180c6670 2024-03-16T19:57:23.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.648+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0730052900 con 0x7f0738158f50 2024-03-16T19:57:23.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.664+0000 7fd9ff7fe640 1 -- 172.21.15.32:0/371878935 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda08055540 con 0x7fda180c6670 2024-03-16T19:57:23.637 INFO:teuthology.orchestra.run.smithi032.stdout:60129542165 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 msgr2=0x7fda0006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 0x7fda0006d860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda10001fe0 tx=0x7fda100015d0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 msgr2=0x7fda18156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda0800b550 tx=0x7fda0800ba10 comp rx=0 tx=0).stop 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 shutdown_connections 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fda0006b3a0 0x7fda0006d860 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda1814fee0 0x7fda18151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda180c6670 0x7fda18156790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 --2- 172.21.15.32:0/371878935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fda1806d540 0x7fda18154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 >> 172.21.15.32:0/371878935 conn(0x7fda1806b190 msgr2=0x7fda180c4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 shutdown_connections 2024-03-16T19:57:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.668+0000 7fda17fff640 1 -- 172.21.15.32:0/371878935 wait complete. 2024-03-16T19:57:23.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542166 got 60129542165 for osd.1 2024-03-16T19:57:23.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.836+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0a6006a340 con 0x7f0a600c68e0 2024-03-16T19:57:23.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.844+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f07381ef140 con 0x7f0738158f50 2024-03-16T19:57:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.852+0000 7f0734ff9640 1 -- 172.21.15.32:0/3134829285 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07381ef140 con 0x7f0738158f50 2024-03-16T19:57:23.824 INFO:teuthology.orchestra.run.smithi032.stdout:68719476756 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 msgr2=0x7f0724069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 0x7f0724069380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b3000 tx=0x7f072c0ae040 comp rx=0 tx=0).stop 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 msgr2=0x7f073814b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0730001340 tx=0x7f0730005e00 comp rx=0 tx=0).stop 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 shutdown_connections 2024-03-16T19:57:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0724066ec0 0x7f0724069380 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0738159da0 0x7f0738144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0738158f50 0x7f073814b010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 --2- 172.21.15.32:0/3134829285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0720003bb0 0x7f073814c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 >> 172.21.15.32:0/3134829285 conn(0x7f073806b190 msgr2=0x7f07380c5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 shutdown_connections 2024-03-16T19:57:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.856+0000 7f073e73a640 1 -- 172.21.15.32:0/3134829285 wait complete. 2024-03-16T19:57:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.876+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fba480ef140 con 0x7fba580c77f0 2024-03-16T19:57:23.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476757 got 68719476756 for osd.4 2024-03-16T19:57:23.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.884+0000 7f0a56ffd640 1 -- 172.21.15.32:0/7559647 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a500b8020 con 0x7f0a600c68e0 2024-03-16T19:57:23.857 INFO:teuthology.orchestra.run.smithi032.stdout:68719476758 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 msgr2=0x7f0a44069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 0x7f0a44069330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a6014a110 tx=0x7f0a5c006bb0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 msgr2=0x7f0a601484a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a50008870 tx=0x7f0a5001fa80 comp rx=0 tx=0).stop 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 shutdown_connections 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a44066e70 0x7f0a44069330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015e950 0x7f0a601edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6014caa0 0x7f0a601571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 --2- 172.21.15.32:0/7559647 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a600c68e0 0x7f0a601484a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 >> 172.21.15.32:0/7559647 conn(0x7f0a6006afc0 msgr2=0x7f0a6014a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7fdeab7fe640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdea400c780 tx=0x7fdea400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 shutdown_connections 2024-03-16T19:57:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.888+0000 7f0a68570640 1 -- 172.21.15.32:0/7559647 wait complete. 2024-03-16T19:57:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.900+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4003d90 con 0x7fdeac1540e0 2024-03-16T19:57:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.900+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac157740 con 0x7fdeac1540e0 2024-03-16T19:57:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.904+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac157d00 con 0x7fdeac1540e0 2024-03-16T19:57:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.904+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea400ce90 con 0x7fdeac1540e0 2024-03-16T19:57:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.904+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea40052b0 con 0x7fdeac1540e0 2024-03-16T19:57:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.904+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900052a0 con 0x7fdeac1540e0 2024-03-16T19:57:23.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476759 got 68719476758 for osd.5 2024-03-16T19:57:23.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f066affd640 1 -- 172.21.15.32:0/177392639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f067001e070 con 0x7f06640047d0 2024-03-16T19:57:23.906 INFO:teuthology.orchestra.run.smithi032.stdout:64424509461 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 msgr2=0x7f065c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 0x7f065c069380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f067c06faf0 tx=0x7f066c004850 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 msgr2=0x7f067c13fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 0x7f067c13fc60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f067000c5a0 tx=0x7f067000ca60 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 shutdown_connections 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f065c066ec0 0x7f065c069380 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f067c15b0e0 0x7f067c139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f067c06f6f0 0x7f067c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 --2- 172.21.15.32:0/177392639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640047d0 0x7f067c13fc60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 >> 172.21.15.32:0/177392639 conn(0x7f067c06b190 msgr2=0x7f067c15d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 shutdown_connections 2024-03-16T19:57:23.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.936+0000 7f0680a14640 1 -- 172.21.15.32:0/177392639 wait complete. 2024-03-16T19:57:23.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509461 for osd.0 2024-03-16T19:57:23.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:23.964+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff320051150 con 0x7ff3280c9e90 2024-03-16T19:57:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.088+0000 7f3f7effd640 1 -- 172.21.15.32:0/3905568258 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700a7a90 con 0x7f3f80150f00 2024-03-16T19:57:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.088+0000 7f3f7effd640 1 -- 172.21.15.32:0/3905568258 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f700a7bf0 con 0x7f3f80150f00 2024-03-16T19:57:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.088+0000 7f3f7effd640 1 -- 172.21.15.32:0/3905568258 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700af440 con 0x7f3f80150f00 2024-03-16T19:57:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- 172.21.15.32:0/3905568258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f80150f00 msgr2=0x7f3f800c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 --2- 172.21.15.32:0/3905568258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f80150f00 0x7f3f800c88d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000bb10 tx=0x7f3f7001ebf0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- 172.21.15.32:0/3905568258 shutdown_connections 2024-03-16T19:57:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 --2- 172.21.15.32:0/3905568258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f800c3540 0x7f3f800c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 --2- 172.21.15.32:0/3905568258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f80150f00 0x7f3f800c88d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- 172.21.15.32:0/3905568258 >> 172.21.15.32:0/3905568258 conn(0x7f3f80066ea0 msgr2=0x7f3f800721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- 172.21.15.32:0/3905568258 shutdown_connections 2024-03-16T19:57:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- 172.21.15.32:0/3905568258 wait complete. 2024-03-16T19:57:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 Processor -- start 2024-03-16T19:57:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.092+0000 7f3f863b3640 1 -- start start 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f800c3540 0x7f3f80151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f80150f00 0x7f3f80154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8015a4c0 con 0x7f3f800c7040 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8015b680 con 0x7f3f800c3540 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f863b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8015a8e0 con 0x7f3f80150f00 2024-03-16T19:57:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f7f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f7f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59726/0 (socket says 172.21.15.32:59726) 2024-03-16T19:57:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.096+0000 7f3f7f7fe640 1 -- 172.21.15.32:0/746915458 learned_addr learned my addr 172.21.15.32:0/746915458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.100+0000 7f3f7f7fe640 1 -- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f80150f00 msgr2=0x7f3f80154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.100+0000 7f3f7f7fe640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f80150f00 0x7f3f80154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.100+0000 7f3f7f7fe640 1 -- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f800c3540 msgr2=0x7f3f80151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.100+0000 7f3f7f7fe640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f800c3540 0x7f3f80151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.100+0000 7f3f7f7fe640 1 -- 172.21.15.32:0/746915458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7000b770 con 0x7f3f800c7040 2024-03-16T19:57:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.104+0000 7f3f7f7fe640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f74001e80 tx=0x7f3f7400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74005480 con 0x7f3f800c7040 2024-03-16T19:57:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8015b3a0 con 0x7f3f800c7040 2024-03-16T19:57:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f80159e00 con 0x7f3f800c7040 2024-03-16T19:57:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f74011680 con 0x7f3f800c7040 2024-03-16T19:57:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74011a50 con 0x7f3f800c7040 2024-03-16T19:57:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.112+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f600052a0 con 0x7f3f800c7040 2024-03-16T19:57:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.136+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdea4005410 con 0x7fdeac1540e0 2024-03-16T19:57:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.136+0000 7fdea37fe640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 0x7fde8c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.136+0000 7fdeaaffd640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 0x7fde8c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.136+0000 7fdeaaffd640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 0x7fde8c069300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0b2000 tx=0x7fde9c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.160+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fba40005820 con 0x7fba580c77f0 2024-03-16T19:57:24.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.164+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff328141fc0 con 0x7ff3280c9e90 2024-03-16T19:57:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.244+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdea40559d0 con 0x7fdeac1540e0 2024-03-16T19:57:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.260+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdea4051570 con 0x7fdeac1540e0 2024-03-16T19:57:24.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba4effd640 1 -- 172.21.15.32:0/316605833 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba480f2800 con 0x7fba580c77f0 2024-03-16T19:57:24.242 INFO:teuthology.orchestra.run.smithi032.stdout:68719476759 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 msgr2=0x7fba3c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 0x7fba3c069380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba54009730 tx=0x7fba54006ca0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 msgr2=0x7fba581fcfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fba4801e3d0 tx=0x7fba4801cac0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 shutdown_connections 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba3c066ec0 0x7fba3c069380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba58150340 0x7fba581f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba580c77f0 0x7fba581fcfa0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 --2- 172.21.15.32:0/316605833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba580c4260 0x7fba58150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 >> 172.21.15.32:0/316605833 conn(0x7fba580c83d0 msgr2=0x7fba580c57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 shutdown_connections 2024-03-16T19:57:24.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.272+0000 7fba6051f640 1 -- 172.21.15.32:0/316605833 wait complete. 2024-03-16T19:57:24.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476760 got 68719476759 for osd.6 2024-03-16T19:57:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.432+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fde90005820 con 0x7fdeac1540e0 2024-03-16T19:57:24.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.440+0000 7fdea37fe640 1 -- 172.21.15.32:0/1484318700 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdea4017020 con 0x7fdeac1540e0 2024-03-16T19:57:24.411 INFO:teuthology.orchestra.run.smithi032.stdout:64424509459 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 msgr2=0x7fde8c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 0x7fde8c069300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0b2000 tx=0x7fde9c004740 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 msgr2=0x7fdeac155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdea400c780 tx=0x7fdea400cc40 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 shutdown_connections 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fde8c066e40 0x7fde8c069300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdeac1540e0 0x7fdeac155240 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac070430 0x7fdeac15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 --2- 172.21.15.32:0/1484318700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdeac06c840 0x7fdeac1fce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 >> 172.21.15.32:0/1484318700 conn(0x7fdeac0cc7a0 msgr2=0x7fdeac072870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 shutdown_connections 2024-03-16T19:57:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.444+0000 7fdeb17a2640 1 -- 172.21.15.32:0/1484318700 wait complete. 2024-03-16T19:57:24.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509460 got 64424509459 for osd.3 2024-03-16T19:57:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.540+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3f74017070 con 0x7f3f800c7040 2024-03-16T19:57:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.540+0000 7f3f7cff9640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 0x7f3f6c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.540+0000 7f3f7ffff640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 0x7f3f6c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.544+0000 7f3f7ffff640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 0x7f3f6c069350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f700b1000 tx=0x7f3f70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.572+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3f7401be60 con 0x7f3f800c7040 2024-03-16T19:57:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.572+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3f74055600 con 0x7f3f800c7040 2024-03-16T19:57:24.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.596+0000 7ff316ffd640 1 -- 172.21.15.32:0/1941148891 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff328141fc0 con 0x7ff3280c9e90 2024-03-16T19:57:24.567 INFO:teuthology.orchestra.run.smithi032.stdout:64424509461 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.596+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 msgr2=0x7ff308069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.596+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 0x7ff308069350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff31801d0e0 tx=0x7ff318001900 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.596+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 msgr2=0x7ff328071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.596+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 0x7ff328071520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff3280c3330 tx=0x7ff32000cb90 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 shutdown_connections 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff308066e90 0x7ff308069350 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3280c9e90 0x7ff328071520 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3280c9a50 0x7ff328070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 --2- 172.21.15.32:0/1941148891 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3100039f0 0x7ff328141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 >> 172.21.15.32:0/1941148891 conn(0x7ff32806b190 msgr2=0x7ff328156690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 shutdown_connections 2024-03-16T19:57:24.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.600+0000 7ff32cf38640 1 -- 172.21.15.32:0/1941148891 wait complete. 2024-03-16T19:57:24.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509461 for osd.2 2024-03-16T19:57:24.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:57:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.760+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3f60005820 con 0x7f3f800c7040 2024-03-16T19:57:24.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.764+0000 7f3f7cff9640 1 -- 172.21.15.32:0/746915458 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f74058cc0 con 0x7f3f800c7040 2024-03-16T19:57:24.735 INFO:teuthology.orchestra.run.smithi032.stdout:68719476757 2024-03-16T19:57:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 msgr2=0x7f3f6c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 0x7f3f6c069350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f700b1000 tx=0x7f3f70008040 comp rx=0 tx=0).stop 2024-03-16T19:57:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 msgr2=0x7f3f8015c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f74001e80 tx=0x7f3f7400c920 comp rx=0 tx=0).stop 2024-03-16T19:57:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 shutdown_connections 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3f6c066e90 0x7f3f6c069350 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3f80150f00 0x7f3f80154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f800c7040 0x7f3f8015c0f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 --2- 172.21.15.32:0/746915458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f800c3540 0x7f3f80151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 >> 172.21.15.32:0/746915458 conn(0x7f3f80066ea0 msgr2=0x7f3f800cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 shutdown_connections 2024-03-16T19:57:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.768+0000 7f3f863b3640 1 -- 172.21.15.32:0/746915458 wait complete. 2024-03-16T19:57:24.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476758 got 68719476757 for osd.7 2024-03-16T19:57:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.860+0000 7f8214142640 1 Processor -- start 2024-03-16T19:57:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.860+0000 7f8214142640 1 -- start start 2024-03-16T19:57:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f820c151070 0x7f820c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c0c3600 0x7f820c0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f820c15a2a0 0x7f820c0c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c151450 con 0x7f820c151070 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c156cb0 con 0x7f820c15a2a0 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8214142640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0ccd70 con 0x7f820c0c3600 2024-03-16T19:57:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8211ede640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f820c151070 0x7f820c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8211ede640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f820c151070 0x7f820c0c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59728/0 (socket says 172.21.15.32:59728) 2024-03-16T19:57:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f8211ede640 1 -- 172.21.15.32:0/1811213999 learned_addr learned my addr 172.21.15.32:0/1811213999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.864+0000 7f82126df640 1 --2- 172.21.15.32:0/1811213999 >> v2:172.21.15.59:3300/0 conn(0x7f820c15a2a0 0x7f820c0c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.868+0000 7f8211ede640 1 -- 172.21.15.32:0/1811213999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c0c3600 msgr2=0x7f820c0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.868+0000 7f8211ede640 1 --2- 172.21.15.32:0/1811213999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c0c3600 0x7f820c0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.868+0000 7f8211ede640 1 -- 172.21.15.32:0/1811213999 >> v2:172.21.15.59:3300/0 conn(0x7f820c15a2a0 msgr2=0x7f820c0c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.868+0000 7f8211ede640 1 --2- 172.21.15.32:0/1811213999 >> v2:172.21.15.59:3300/0 conn(0x7f820c15a2a0 0x7f820c0c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.868+0000 7f8211ede640 1 -- 172.21.15.32:0/1811213999 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0cd050 con 0x7f820c151070 2024-03-16T19:57:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.872+0000 7f8211ede640 1 --2- 172.21.15.32:0/1811213999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c0c70f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f820c0c78d0 tx=0x7f820801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fc5ef82b8f217f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8210edc640 1 -- 172.21.15.32:0/1811213999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080ac630 con 0x7f820c151070 2024-03-16T19:57:24.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8210edc640 1 -- 172.21.15.32:0/1811213999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82080ac790 con 0x7f820c151070 2024-03-16T19:57:24.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8210edc640 1 -- 172.21.15.32:0/1811213999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080acb20 con 0x7f820c151070 2024-03-16T19:57:24.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 -- 172.21.15.32:0/1811213999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 msgr2=0x7f820c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 --2- 172.21.15.32:0/1811213999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c0c70f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f820c0c78d0 tx=0x7f820801c520 comp rx=0 tx=0).stop 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 -- 172.21.15.32:0/1811213999 shutdown_connections 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 --2- 172.21.15.32:0/1811213999 >> v2:172.21.15.59:3300/0 conn(0x7f820c15a2a0 0x7f820c0c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 --2- 172.21.15.32:0/1811213999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c0c3600 0x7f820c0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 --2- 172.21.15.32:0/1811213999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c0c70f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 -- 172.21.15.32:0/1811213999 >> 172.21.15.32:0/1811213999 conn(0x7f820c06e6c0 msgr2=0x7f820c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 -- 172.21.15.32:0/1811213999 shutdown_connections 2024-03-16T19:57:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.876+0000 7f8214142640 1 -- 172.21.15.32:0/1811213999 wait complete. 2024-03-16T19:57:24.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:57:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 Processor -- start 2024-03-16T19:57:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 -- start start 2024-03-16T19:57:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0c3600 0x7f820c142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c15a2a0 0x7f820c1f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c1493f0 con 0x7f820c151070 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c150380 con 0x7f820c0c3600 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8214142640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c1f0b20 con 0x7f820c15a2a0 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f82116dd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f82116dd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59744/0 (socket says 172.21.15.32:59744) 2024-03-16T19:57:24.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f82116dd640 1 -- 172.21.15.32:0/2172681421 learned_addr learned my addr 172.21.15.32:0/2172681421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:24.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.880+0000 7f8211ede640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0c3600 0x7f820c142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.884+0000 7f82116dd640 1 -- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c15a2a0 msgr2=0x7f820c1f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.884+0000 7f82116dd640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c15a2a0 0x7f820c1f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.884+0000 7f82116dd640 1 -- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0c3600 msgr2=0x7f820c142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.884+0000 7f82116dd640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0c3600 0x7f820c142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.884+0000 7f82116dd640 1 -- 172.21.15.32:0/2172681421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080094d0 con 0x7f820c151070 2024-03-16T19:57:24.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.888+0000 7f82116dd640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b480 tx=0x7f81fc00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc011830 con 0x7f820c151070 2024-03-16T19:57:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc011990 con 0x7f820c151070 2024-03-16T19:57:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c14a230 con 0x7f820c151070 2024-03-16T19:57:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc011d40 con 0x7f820c151070 2024-03-16T19:57:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c1f0de0 con 0x7f820c151070 2024-03-16T19:57:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.892+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c068000 con 0x7f820c151070 2024-03-16T19:57:24.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:57:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.924+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f81fc01a400 con 0x7f820c151070 2024-03-16T19:57:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.924+0000 7f82027fc640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 0x7f81f0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.924+0000 7f8211ede640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 0x7f81f0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.924+0000 7f8211ede640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 0x7f81f0069350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f82080ae000 tx=0x7f82080a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.928+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f81fc056330 con 0x7f820c151070 2024-03-16T19:57:24.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:57:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.996+0000 7efc9551a640 1 Processor -- start 2024-03-16T19:57:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.996+0000 7efc9551a640 1 -- start start 2024-03-16T19:57:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:24.996+0000 7efc9551a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 0x7efc90072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc9551a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9015abe0 con 0x7efc9015d4c0 2024-03-16T19:57:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc9551a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc90070b90 con 0x7efc90070430 2024-03-16T19:57:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc9551a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc90159fd0 con 0x7efc901540e0 2024-03-16T19:57:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc8effd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 0x7efc90072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc8e7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efc90070430 0x7efc9006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52614/0 (socket says 172.21.15.32:52614) 2024-03-16T19:57:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.000+0000 7efc8e7fc640 1 -- 172.21.15.32:0/220426980 learned_addr learned my addr 172.21.15.32:0/220426980 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.004+0000 7efc8effd640 1 -- 172.21.15.32:0/220426980 >> v1:172.21.15.59:6789/0 conn(0x7efc90070430 legacy=0x7efc9006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.004+0000 7efc8effd640 1 -- 172.21.15.32:0/220426980 >> v1:172.21.15.32:6789/0 conn(0x7efc9015d4c0 legacy=0x7efc901647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.004+0000 7efc8effd640 1 -- 172.21.15.32:0/220426980 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9015a2b0 con 0x7efc901540e0 2024-03-16T19:57:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.008+0000 7efc8effd640 1 --2- 172.21.15.32:0/220426980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 0x7efc90072b30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efc800097f0 tx=0x7efc8001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789b36bcc9e24d97 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:24.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc8dffb640 1 -- 172.21.15.32:0/220426980 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc800a8990 con 0x7efc901540e0 2024-03-16T19:57:24.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc8dffb640 1 -- 172.21.15.32:0/220426980 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc800a8af0 con 0x7efc901540e0 2024-03-16T19:57:24.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc8dffb640 1 -- 172.21.15.32:0/220426980 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc800a8e80 con 0x7efc901540e0 2024-03-16T19:57:24.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 -- 172.21.15.32:0/220426980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 msgr2=0x7efc90072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 --2- 172.21.15.32:0/220426980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 0x7efc90072b30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efc800097f0 tx=0x7efc8001ec90 comp rx=0 tx=0).stop 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 -- 172.21.15.32:0/220426980 shutdown_connections 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 --2- 172.21.15.32:0/220426980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc901540e0 0x7efc90072b30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 -- 172.21.15.32:0/220426980 >> 172.21.15.32:0/220426980 conn(0x7efc900cc7a0 msgr2=0x7efc900ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 -- 172.21.15.32:0/220426980 shutdown_connections 2024-03-16T19:57:24.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.016+0000 7efc9551a640 1 -- 172.21.15.32:0/220426980 wait complete. 2024-03-16T19:57:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 Processor -- start 2024-03-16T19:57:24.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 -- start start 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc90070430 0x7efc9014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc9014eb60 0x7efc901eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9014d2c0 con 0x7efc90070430 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9014e6d0 con 0x7efc9014eb60 2024-03-16T19:57:24.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc9551a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9014e0c0 con 0x7efc9015d4c0 2024-03-16T19:57:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc8e7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc8f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc9014eb60 0x7efc901eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc8e7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43988/0 (socket says 172.21.15.32:43988) 2024-03-16T19:57:24.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.020+0000 7efc8e7fc640 1 -- 172.21.15.32:0/1476019042 learned_addr learned my addr 172.21.15.32:0/1476019042 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.028+0000 7efc8e7fc640 1 -- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc9014eb60 msgr2=0x7efc901eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.028+0000 7efc8e7fc640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc9014eb60 0x7efc901eff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.028+0000 7efc8e7fc640 1 -- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc90070430 msgr2=0x7efc9014ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.028+0000 7efc8e7fc640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc90070430 0x7efc9014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.028+0000 7efc8e7fc640 1 -- 172.21.15.32:0/1476019042 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc80009420 con 0x7efc9015d4c0 2024-03-16T19:57:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.032+0000 7efc8e7fc640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc8400c530 tx=0x7efc8400c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84005cf0 con 0x7efc9015d4c0 2024-03-16T19:57:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc84005e50 con 0x7efc9015d4c0 2024-03-16T19:57:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84007270 con 0x7efc9015d4c0 2024-03-16T19:57:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc9551a640 1 -- 172.21.15.32:0/1476019042 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9014ca50 con 0x7efc9015d4c0 2024-03-16T19:57:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc9551a640 1 -- 172.21.15.32:0/1476019042 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9014cfe0 con 0x7efc9015d4c0 2024-03-16T19:57:25.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.040+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc700052a0 con 0x7efc9015d4c0 2024-03-16T19:57:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.052+0000 7feddd63e640 1 Processor -- start 2024-03-16T19:57:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.052+0000 7feddd63e640 1 -- start start 2024-03-16T19:57:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.056+0000 7feddd63e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd80c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.056+0000 7feddd63e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedd8073110 0x7fedd815e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.056+0000 7feddd63e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd80c3830 con 0x7fedd815e910 2024-03-16T19:57:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.056+0000 7feddd63e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80723b0 con 0x7fedd8073110 2024-03-16T19:57:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.056+0000 7feddd63e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81627a0 con 0x7fedd8158700 2024-03-16T19:57:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.080+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc840073d0 con 0x7efc9015d4c0 2024-03-16T19:57:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.080+0000 7efc777fe640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 0x7efc7c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.080+0000 7efc8effd640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 0x7efc7c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.080+0000 7efc8effd640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 0x7efc7c069350 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc800b2000 tx=0x7efc80004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 Processor -- start 2024-03-16T19:57:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 -- start start 2024-03-16T19:57:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 0x7f89b8070430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b815d760 con 0x7f89b815ab20 2024-03-16T19:57:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b815da40 con 0x7f89b80725b0 2024-03-16T19:57:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89bcb45640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b815dd20 con 0x7f89b8073180 2024-03-16T19:57:25.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89b659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f89b80725b0 0x7f89b815a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52622/0 (socket says 172.21.15.32:52622) 2024-03-16T19:57:25.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.100+0000 7f89b659c640 1 -- 172.21.15.32:0/651881392 learned_addr learned my addr 172.21.15.32:0/651881392 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:25.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.108+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324856998 0 0) 0x7f89b815da40 con 0x7f89b80725b0 2024-03-16T19:57:25.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.108+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a0003530 con 0x7f89b80725b0 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091176934 0 0) 0x7f89a0003530 con 0x7f89b80725b0 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 msgr2=0x7f89b8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 0x7f89b8070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 >> v1:172.21.15.32:6789/0 conn(0x7f89b815ab20 legacy=0x7f89b81627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b815e000 con 0x7f89b80725b0 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f89a8001680 con 0x7f89b80725b0 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 >> v1:172.21.15.59:6789/0 conn(0x7f89b80725b0 legacy=0x7f89b815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 0x7f89a0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a00040c0 0x7f89a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89a0004e30 0x7f89a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b815dd20 con 0x7f89a00040c0 2024-03-16T19:57:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a00055a0 con 0x7f89a0004e30 2024-03-16T19:57:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a000bed0 con 0x7f89b8073180 2024-03-16T19:57:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.120+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89a0004e30 0x7f89a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.128+0000 7f89b6d9d640 1 -- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 msgr2=0x7f89a0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.128+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b8073180 0x7f89a0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.128+0000 7f89b6d9d640 1 -- 172.21.15.32:0/651881392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a00040c0 msgr2=0x7f89a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.128+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a00040c0 0x7f89a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.128+0000 7f89b6d9d640 1 -- 172.21.15.32:0/651881392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0003530 con 0x7f89a0004e30 2024-03-16T19:57:25.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.136+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89a0004e30 0x7f89a000b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009ca0 tx=0x7f89b001cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2954e61e6854f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.140+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b00a5c80 con 0x7f89a0004e30 2024-03-16T19:57:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89b559a640 1 -- 172.21.15.32:0/651881392 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b00a5e00 con 0x7f89a0004e30 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89a0004e30 msgr2=0x7f89a000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 --2- 172.21.15.32:0/651881392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89a0004e30 0x7f89a000b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009ca0 tx=0x7f89b001cc70 comp rx=0 tx=0).stop 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89b6d9d640 1 -- 172.21.15.32:0/651881392 reap_dead start 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- 172.21.15.32:0/651881392 shutdown_connections 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- 172.21.15.32:0/651881392 >> 172.21.15.32:0/651881392 conn(0x7f89b80cc7a0 msgr2=0x7f89b80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- 172.21.15.32:0/651881392 shutdown_connections 2024-03-16T19:57:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- 172.21.15.32:0/651881392 wait complete. 2024-03-16T19:57:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 Processor -- start 2024-03-16T19:57:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.144+0000 7f89bcb45640 1 -- start start 2024-03-16T19:57:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b8147980 0x7f89b814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b814c980 0x7f89b81e5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b814e570 con 0x7f89b8147980 2024-03-16T19:57:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b814d500 con 0x7f89b81e5aa0 2024-03-16T19:57:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89bcb45640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8147d60 con 0x7f89b814c980 2024-03-16T19:57:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89b6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89b6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43866/0 (socket says 172.21.15.32:43866) 2024-03-16T19:57:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.148+0000 7f89b6d9d640 1 -- 172.21.15.32:0/1458256373 learned_addr learned my addr 172.21.15.32:0/1458256373 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.160+0000 7f89b6d9d640 1 -- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b814c980 msgr2=0x7f89b81e5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.160+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b814c980 0x7f89b81e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.160+0000 7f89b6d9d640 1 -- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b8147980 msgr2=0x7f89b814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.160+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b8147980 0x7f89b814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.160+0000 7f89b6d9d640 1 -- 172.21.15.32:0/1458256373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b80c35c0 con 0x7f89b81e5aa0 2024-03-16T19:57:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.164+0000 7f89b6d9d640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89b00987a0 tx=0x7f89a0005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.172+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b00bd070 con 0x7f89b81e5aa0 2024-03-16T19:57:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.176+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b00a5c90 con 0x7f89b81e5aa0 2024-03-16T19:57:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.176+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b814d240 con 0x7f89b81e5aa0 2024-03-16T19:57:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.176+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b00b8660 con 0x7f89b81e5aa0 2024-03-16T19:57:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.176+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b80ca500 con 0x7f89b81e5aa0 2024-03-16T19:57:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.176+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b8070b20 con 0x7f89b81e5aa0 2024-03-16T19:57:25.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.248+0000 7fedd67fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedd8073110 0x7fedd815e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.248+0000 7fedd77fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedd815e910 0x7fedd80725a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48986/0 (socket says 172.21.15.32:48986) 2024-03-16T19:57:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.248+0000 7fedd77fe640 1 -- 172.21.15.32:0/3349424326 learned_addr learned my addr 172.21.15.32:0/3349424326 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.268+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f81fc051ed0 con 0x7f820c151070 2024-03-16T19:57:25.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.288+0000 7fedd67fc640 1 -- 172.21.15.32:0/3349424326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 msgr2=0x7fedd80c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.288+0000 7fedd67fc640 1 --2- 172.21.15.32:0/3349424326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd80c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.288+0000 7fedd67fc640 1 -- 172.21.15.32:0/3349424326 >> v1:172.21.15.32:6789/0 conn(0x7fedd815e910 legacy=0x7fedd80725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.288+0000 7fedd67fc640 1 -- 172.21.15.32:0/3349424326 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8070ae0 con 0x7fedd8073110 2024-03-16T19:57:25.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:57:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.304+0000 7fedd67fc640 1 --2- 172.21.15.32:0/3349424326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8073110 0x7fedd815e3d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0097c0 tx=0x7fedcc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5f2c0231096699 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7fedd5ffb640 1 -- 172.21.15.32:0/3349424326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0a8990 con 0x7fedd8073110 2024-03-16T19:57:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7fedd5ffb640 1 -- 172.21.15.32:0/3349424326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc0a8af0 con 0x7fedd8073110 2024-03-16T19:57:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7fedd5ffb640 1 -- 172.21.15.32:0/3349424326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0a8e80 con 0x7fedd8073110 2024-03-16T19:57:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 -- 172.21.15.32:0/3349424326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8073110 msgr2=0x7fedd815e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 --2- 172.21.15.32:0/3349424326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8073110 0x7fedd815e3d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0097c0 tx=0x7fedcc01ec90 comp rx=0 tx=0).stop 2024-03-16T19:57:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 -- 172.21.15.32:0/3349424326 shutdown_connections 2024-03-16T19:57:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 --2- 172.21.15.32:0/3349424326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8073110 0x7fedd815e3d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 --2- 172.21.15.32:0/3349424326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd80c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 -- 172.21.15.32:0/3349424326 >> 172.21.15.32:0/3349424326 conn(0x7fedd80cc7a0 msgr2=0x7fedd80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 -- 172.21.15.32:0/3349424326 shutdown_connections 2024-03-16T19:57:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.308+0000 7feddd63e640 1 -- 172.21.15.32:0/3349424326 wait complete. 2024-03-16T19:57:25.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 Processor -- start 2024-03-16T19:57:25.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 -- start start 2024-03-16T19:57:25.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedd8073110 0x7fedd81f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd8160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd815f9d0 con 0x7fedd815e910 2024-03-16T19:57:25.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8160b10 con 0x7fedd8073110 2024-03-16T19:57:25.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.312+0000 7feddd63e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd815fc20 con 0x7fedd8158700 2024-03-16T19:57:25.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.316+0000 7fedd77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.316+0000 7fedd77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59788/0 (socket says 172.21.15.32:59788) 2024-03-16T19:57:25.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.316+0000 7fedd77fe640 1 -- 172.21.15.32:0/1510654823 learned_addr learned my addr 172.21.15.32:0/1510654823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.320+0000 7fedd77fe640 1 -- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 msgr2=0x7fedd8160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.320+0000 7fedd77fe640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd8160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.320+0000 7fedd77fe640 1 -- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedd8073110 msgr2=0x7fedd81f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.320+0000 7fedd77fe640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedd8073110 0x7fedd81f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.320+0000 7fedd77fe640 1 -- 172.21.15.32:0/1510654823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc009420 con 0x7fedd815e910 2024-03-16T19:57:25.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.324+0000 7fedd77fe640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fedd000ca20 tx=0x7fedd000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.328+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd000abd0 con 0x7fedd815e910 2024-03-16T19:57:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.332+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd000ad30 con 0x7fedd815e910 2024-03-16T19:57:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.332+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd815ff00 con 0x7fedd815e910 2024-03-16T19:57:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.332+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0012620 con 0x7fedd815e910 2024-03-16T19:57:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.332+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd81604c0 con 0x7fedd815e910 2024-03-16T19:57:25.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.332+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd806cc30 con 0x7fedd815e910 2024-03-16T19:57:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.368+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fedd0007070 con 0x7fedd815e910 2024-03-16T19:57:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.368+0000 7fedbf7fe640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 0x7fedc4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.368+0000 7fedd6ffd640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 0x7fedc4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.368+0000 7fedd6ffd640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 0x7fedc4069350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009690 tx=0x7fedc8006c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.384+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efc84054f60 con 0x7efc9015d4c0 2024-03-16T19:57:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 Processor -- start 2024-03-16T19:57:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 -- start start 2024-03-16T19:57:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ea415e070 0x7f2ea4071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ea415d570 0x7f2ea415da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea40c30e0 con 0x7f2ea4154560 2024-03-16T19:57:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea40c2320 con 0x7f2ea415d570 2024-03-16T19:57:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7f2eab5a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea4073500 con 0x7f2ea415e070 2024-03-16T19:57:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.408+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efc84051200 con 0x7efc9015d4c0 2024-03-16T19:57:25.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-16T19:57:25.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.496+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f820c14adb0 con 0x7f820c151070 2024-03-16T19:57:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.496+0000 7f2ea8b3f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ea415e070 0x7f2ea4071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.496+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f89b00b4040 con 0x7f89b81e5aa0 2024-03-16T19:57:25.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.500+0000 7f89a6ffd640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 0x7f8998069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.500+0000 7f89b659c640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 0x7f8998069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.500+0000 7f89b659c640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 0x7f8998069350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f89a8007a80 tx=0x7f89a8001560 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.504+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f89b00effd0 con 0x7f89b81e5aa0 2024-03-16T19:57:25.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.528+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f89b00ebb70 con 0x7f89b81e5aa0 2024-03-16T19:57:25.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.532+0000 7f2ea8b3f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ea415e070 0x7f2ea4071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43892/0 (socket says 172.21.15.32:43892) 2024-03-16T19:57:25.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.532+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3683484950 learned_addr learned my addr 172.21.15.32:0/3683484950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.544+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3683484950 >> v2:172.21.15.32:3300/0 conn(0x7f2ea415d570 msgr2=0x7f2ea415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.544+0000 7f2ea8b3f640 1 --2- 172.21.15.32:0/3683484950 >> v2:172.21.15.32:3300/0 conn(0x7f2ea415d570 0x7f2ea415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.544+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3683484950 >> v1:172.21.15.32:6789/0 conn(0x7f2ea4154560 legacy=0x7f2ea40c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.544+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3683484950 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4073220 con 0x7f2ea415e070 2024-03-16T19:57:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.556+0000 7f2ea8b3f640 1 --2- 172.21.15.32:0/3683484950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415e070 0x7f2ea4071970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4067530 tx=0x7f2e9401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b281975769ea4a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2e9bfff640 1 -- 172.21.15.32:0/3683484950 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940aa610 con 0x7f2ea415e070 2024-03-16T19:57:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2e9bfff640 1 -- 172.21.15.32:0/3683484950 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e940a6040 con 0x7f2ea415e070 2024-03-16T19:57:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2e9bfff640 1 -- 172.21.15.32:0/3683484950 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940aa990 con 0x7f2ea415e070 2024-03-16T19:57:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3683484950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415e070 msgr2=0x7f2ea4071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3683484950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415e070 0x7f2ea4071970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4067530 tx=0x7f2e9401c4b0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3683484950 shutdown_connections 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3683484950 >> v2:172.21.15.32:3300/0 conn(0x7f2ea415d570 0x7f2ea415da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3683484950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415e070 0x7f2ea4071970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3683484950 >> 172.21.15.32:0/3683484950 conn(0x7f2ea406f3a0 msgr2=0x7f2ea406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3683484950 shutdown_connections 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3683484950 wait complete. 2024-03-16T19:57:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.568+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fedd0055e00 con 0x7fedd815e910 2024-03-16T19:57:25.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 Processor -- start 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 -- start start 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ea4154560 0x7f2ea414bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ea415e070 0x7f2ea4148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4155ce0 con 0x7f2ea415e070 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea415c9d0 con 0x7f2ea415d570 2024-03-16T19:57:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2eab5a4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41f71d0 con 0x7f2ea4154560 2024-03-16T19:57:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2ea9340640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ea4154560 0x7f2ea414bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2ea8b3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2ea8b3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43896/0 (socket says 172.21.15.32:43896) 2024-03-16T19:57:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.572+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3055621350 learned_addr learned my addr 172.21.15.32:0/3055621350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ea4154560 msgr2=0x7f2ea414bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f2ea8b3f640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ea4154560 0x7f2ea414bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ea415e070 msgr2=0x7f2ea4148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f2ea8b3f640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ea415e070 0x7f2ea4148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f2ea8b3f640 1 -- 172.21.15.32:0/3055621350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94009480 con 0x7f2ea415d570 2024-03-16T19:57:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f4c8faf7640 1 Processor -- start 2024-03-16T19:57:25.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f4c8faf7640 1 -- start start 2024-03-16T19:57:25.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f4c8faf7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c880705b0 0x7f4c880c8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.580+0000 7f4c8faf7640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880cdc60 con 0x7f4c88151fc0 2024-03-16T19:57:25.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.584+0000 7f4c8faf7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880cdf40 con 0x7f4c880c8d00 2024-03-16T19:57:25.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.584+0000 7f4c8faf7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880ce500 con 0x7f4c880705b0 2024-03-16T19:57:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.584+0000 7f2ea8b3f640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e9401a0e0 tx=0x7f2e94019250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94003cf0 con 0x7f2ea415d570 2024-03-16T19:57:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e94003e50 con 0x7f2ea415d570 2024-03-16T19:57:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940aa650 con 0x7f2ea415d570 2024-03-16T19:57:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea40c8060 con 0x7f2ea415d570 2024-03-16T19:57:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea41f68a0 con 0x7f2ea415d570 2024-03-16T19:57:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f4c8e094640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4c880c8d00 0x7f4c881626b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52624/0 (socket says 172.21.15.32:52624) 2024-03-16T19:57:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.592+0000 7f4c8e094640 1 -- 172.21.15.32:0/3045378908 learned_addr learned my addr 172.21.15.32:0/3045378908 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.596+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea41f7340 con 0x7f2ea415d570 2024-03-16T19:57:25.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:57:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.708+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efc70005820 con 0x7efc9015d4c0 2024-03-16T19:57:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc777fe640 1 -- 172.21.15.32:0/1476019042 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc84021020 con 0x7efc9015d4c0 2024-03-16T19:57:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:68719476756 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 msgr2=0x7efc7c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 0x7efc7c069350 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc800b2000 tx=0x7efc80004740 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 msgr2=0x7efc9014c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc8400c530 tx=0x7efc8400c9f0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 shutdown_connections 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efc7c066e90 0x7efc7c069350 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.716+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc9014eb60 0x7efc901eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.720+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efc9015d4c0 0x7efc9014c510 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.720+0000 7efc757fa640 1 --2- 172.21.15.32:0/1476019042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc90070430 0x7efc9014ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.720+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 >> 172.21.15.32:0/1476019042 conn(0x7efc900cc7a0 msgr2=0x7efc9006f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.720+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 shutdown_connections 2024-03-16T19:57:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.720+0000 7efc757fa640 1 -- 172.21.15.32:0/1476019042 wait complete. 2024-03-16T19:57:25.713 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476757 got 68719476756 for osd.4 2024-03-16T19:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.740+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fedd00519a0 con 0x7fedd815e910 2024-03-16T19:57:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 Processor -- start 2024-03-16T19:57:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 -- start start 2024-03-16T19:57:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ca80725b0 0x7f1ca8070430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca815d760 con 0x7f1ca815a410 2024-03-16T19:57:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca8072ed0 con 0x7f1ca80725b0 2024-03-16T19:57:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1cad7ad640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca815da40 con 0x7f1ca8073180 2024-03-16T19:57:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1c9ffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1c9ffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44064/0 (socket says 172.21.15.32:44064) 2024-03-16T19:57:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.768+0000 7f1c9ffff640 1 -- 172.21.15.32:0/3645564411 learned_addr learned my addr 172.21.15.32:0/3645564411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.772+0000 7f1c9ffff640 1 -- 172.21.15.32:0/3645564411 >> v2:172.21.15.32:3300/0 conn(0x7f1ca80725b0 msgr2=0x7f1ca8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.772+0000 7f1c9ffff640 1 --2- 172.21.15.32:0/3645564411 >> v2:172.21.15.32:3300/0 conn(0x7f1ca80725b0 0x7f1ca8070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.772+0000 7f1c9ffff640 1 -- 172.21.15.32:0/3645564411 >> v1:172.21.15.59:6789/0 conn(0x7f1ca815a410 legacy=0x7f1ca815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.772+0000 7f1c9ffff640 1 -- 172.21.15.32:0/3645564411 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca815dd20 con 0x7f1ca8073180 2024-03-16T19:57:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.780+0000 7f1c9ffff640 1 --2- 172.21.15.32:0/3645564411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009870 tx=0x7f1c9001ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0d185a173f559cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.784+0000 7f1ca67fc640 1 -- 172.21.15.32:0/3645564411 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900a89d0 con 0x7f1ca8073180 2024-03-16T19:57:25.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-16T19:57:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1ca67fc640 1 -- 172.21.15.32:0/3645564411 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c900a8b30 con 0x7f1ca8073180 2024-03-16T19:57:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1ca67fc640 1 -- 172.21.15.32:0/3645564411 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900b0440 con 0x7f1ca8073180 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 -- 172.21.15.32:0/3645564411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 msgr2=0x7f1ca8072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/3645564411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009870 tx=0x7f1c9001ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 -- 172.21.15.32:0/3645564411 shutdown_connections 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/3645564411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca8073180 0x7f1ca8072990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/3645564411 >> v2:172.21.15.32:3300/0 conn(0x7f1ca80725b0 0x7f1ca8070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 -- 172.21.15.32:0/3645564411 >> 172.21.15.32:0/3645564411 conn(0x7f1ca80cc7a0 msgr2=0x7f1ca80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 -- 172.21.15.32:0/3645564411 shutdown_connections 2024-03-16T19:57:25.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.788+0000 7f1cad7ad640 1 -- 172.21.15.32:0/3645564411 wait complete. 2024-03-16T19:57:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 Processor -- start 2024-03-16T19:57:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 -- start start 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ca8073180 0x7f1ca81580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ca815a410 0x7f1ca8158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca81f0f60 con 0x7f1ca815a410 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca8152ad0 con 0x7f1ca8073180 2024-03-16T19:57:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1cad7ad640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca80cc080 con 0x7f1ca80725b0 2024-03-16T19:57:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1ca6ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1ca6ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44070/0 (socket says 172.21.15.32:44070) 2024-03-16T19:57:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.792+0000 7f1ca6ffd640 1 -- 172.21.15.32:0/2720252724 learned_addr learned my addr 172.21.15.32:0/2720252724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.796+0000 7f1ca6ffd640 1 -- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ca8073180 msgr2=0x7f1ca81580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.800+0000 7f1ca6ffd640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ca8073180 0x7f1ca81580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.800+0000 7f1ca6ffd640 1 -- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ca815a410 msgr2=0x7f1ca8158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.800+0000 7f1ca6ffd640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ca815a410 0x7f1ca8158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.800+0000 7f1ca6ffd640 1 -- 172.21.15.32:0/2720252724 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c900094d0 con 0x7f1ca80725b0 2024-03-16T19:57:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.800+0000 7f1ca6ffd640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b580 tx=0x7f1c9800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98010870 con 0x7f1ca80725b0 2024-03-16T19:57:25.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca80cc320 con 0x7f1ca80725b0 2024-03-16T19:57:25.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca81f05c0 con 0x7f1ca80725b0 2024-03-16T19:57:25.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c980109f0 con 0x7f1ca80725b0 2024-03-16T19:57:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98010da0 con 0x7f1ca80725b0 2024-03-16T19:57:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.808+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c0052a0 con 0x7f1ca80725b0 2024-03-16T19:57:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.832+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f89b8070b20 con 0x7f89b81e5aa0 2024-03-16T19:57:25.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.848+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1c98023480 con 0x7f1ca80725b0 2024-03-16T19:57:25.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.848+0000 7f4c8d092640 1 --2- 172.21.15.32:0/3045378908 >> v2:172.21.15.32:3300/0 conn(0x7f4c880705b0 0x7f4c880c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.848+0000 7f1c9f7fe640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 0x7f1c88069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.848+0000 7f1c9ffff640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 0x7f1c88069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.848+0000 7f1c9ffff640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 0x7f1c88069350 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1c900b2000 tx=0x7f1c90004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.852+0000 7f4c8d092640 1 -- 172.21.15.32:0/3045378908 >> v1:172.21.15.59:6789/0 conn(0x7f4c880c8d00 legacy=0x7f4c881626b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.852+0000 7f4c8d092640 1 -- 172.21.15.32:0/3045378908 >> v1:172.21.15.32:6789/0 conn(0x7f4c88151fc0 legacy=0x7f4c8815b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.852+0000 7f4c8d092640 1 -- 172.21.15.32:0/3045378908 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880ce220 con 0x7f4c880705b0 2024-03-16T19:57:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.868+0000 7f4c8d092640 1 --2- 172.21.15.32:0/3045378908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c880705b0 0x7f4c880c8100 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c780097c0 tx=0x7f4c7801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef47b98edec5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8c891640 1 -- 172.21.15.32:0/3045378908 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c780a8950 con 0x7f4c880705b0 2024-03-16T19:57:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8c891640 1 -- 172.21.15.32:0/3045378908 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c780a8ab0 con 0x7f4c880705b0 2024-03-16T19:57:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8c891640 1 -- 172.21.15.32:0/3045378908 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c780a8e40 con 0x7f4c880705b0 2024-03-16T19:57:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 -- 172.21.15.32:0/3045378908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c880705b0 msgr2=0x7f4c880c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/3045378908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c880705b0 0x7f4c880c8100 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c780097c0 tx=0x7f4c7801ec50 comp rx=0 tx=0).stop 2024-03-16T19:57:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 -- 172.21.15.32:0/3045378908 shutdown_connections 2024-03-16T19:57:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/3045378908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c880705b0 0x7f4c880c8100 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 -- 172.21.15.32:0/3045378908 >> 172.21.15.32:0/3045378908 conn(0x7f4c88066ea0 msgr2=0x7f4c880672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 -- 172.21.15.32:0/3045378908 shutdown_connections 2024-03-16T19:57:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.892+0000 7f4c8faf7640 1 -- 172.21.15.32:0/3045378908 wait complete. 2024-03-16T19:57:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f82027fc640 1 -- 172.21.15.32:0/2172681421 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81fc015090 con 0x7f820c151070 2024-03-16T19:57:25.867 INFO:teuthology.orchestra.run.smithi032.stdout:60129542165 2024-03-16T19:57:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 Processor -- start 2024-03-16T19:57:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 -- start start 2024-03-16T19:57:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c880705b0 0x7f4c88154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c880c8d00 0x7f4c880c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88155460 con 0x7f4c88151fc0 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8814d8c0 con 0x7f4c880705b0 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8faf7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88154210 con 0x7f4c880c8d00 2024-03-16T19:57:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.896+0000 7f4c8e094640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 msgr2=0x7f81f0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 0x7f81f0069350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f82080ae000 tx=0x7f82080a7000 comp rx=0 tx=0).stop 2024-03-16T19:57:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 msgr2=0x7f820c14fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b480 tx=0x7f81fc00b940 comp rx=0 tx=0).stop 2024-03-16T19:57:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 shutdown_connections 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f81f0066e90 0x7f81f0069350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f820c15a2a0 0x7f820c1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c151070 0x7f820c14fe40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 --2- 172.21.15.32:0/2172681421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f820c0c3600 0x7f820c142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 >> 172.21.15.32:0/2172681421 conn(0x7f820c06e6c0 msgr2=0x7f820c0c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 shutdown_connections 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f8214142640 1 -- 172.21.15.32:0/2172681421 wait complete. 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f4c8e094640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59826/0 (socket says 172.21.15.32:59826) 2024-03-16T19:57:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.900+0000 7f4c8e094640 1 -- 172.21.15.32:0/1496667451 learned_addr learned my addr 172.21.15.32:0/1496667451 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.908+0000 7f4c8e094640 1 -- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c880c8d00 msgr2=0x7f4c880c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.908+0000 7f4c8e094640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c880c8d00 0x7f4c880c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.908+0000 7f4c8e094640 1 -- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c880705b0 msgr2=0x7f4c88154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.908+0000 7f4c8e094640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c880705b0 0x7f4c88154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.908+0000 7f4c8e094640 1 -- 172.21.15.32:0/1496667451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c78009420 con 0x7f4c88151fc0 2024-03-16T19:57:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 Processor -- start 2024-03-16T19:57:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 -- start start 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec0c68e0 0x7fa1ec0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1ec0c87b0 0x7fa1ec160960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec156a60 con 0x7fa1ec14caa0 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec14ce80 con 0x7fa1ec0c87b0 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f4598640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec156c90 con 0x7fa1ec0c68e0 2024-03-16T19:57:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f2b35640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1ec0c87b0 0x7fa1ec160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f2b35640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1ec0c87b0 0x7fa1ec160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59840/0 (socket says 172.21.15.32:59840) 2024-03-16T19:57:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.912+0000 7fa1f2b35640 1 -- 172.21.15.32:0/517950401 learned_addr learned my addr 172.21.15.32:0/517950401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.916+0000 7fa1f2b35640 1 -- 172.21.15.32:0/517950401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec0c68e0 msgr2=0x7fa1ec0c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.916+0000 7fa1f2b35640 1 --2- 172.21.15.32:0/517950401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec0c68e0 0x7fa1ec0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.916+0000 7fa1f2b35640 1 -- 172.21.15.32:0/517950401 >> v1:172.21.15.32:6789/0 conn(0x7fa1ec14caa0 legacy=0x7fa1ec159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.916+0000 7fa1f2b35640 1 -- 172.21.15.32:0/517950401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec156f30 con 0x7fa1ec0c87b0 2024-03-16T19:57:25.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7f4c8e094640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000caa0 tx=0x7f4c8000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2e940aa7b0 con 0x7f2ea415d570 2024-03-16T19:57:25.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7f2e99ffb640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 0x7f2e88069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:25.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7f2ea9340640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 0x7f2e88069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:25.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7f2ea9340640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 0x7f2e88069330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0007fd0 tx=0x7f2ea0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542166 got 60129542165 for osd.1 2024-03-16T19:57:25.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.920+0000 7fa1f2b35640 1 --2- 172.21.15.32:0/517950401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec160960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009e50 tx=0x7fa1e401f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1ded4ebd192400 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:25.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.924+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8000ac00 con 0x7f4c88151fc0 2024-03-16T19:57:25.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.924+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8000ad60 con 0x7f4c88151fc0 2024-03-16T19:57:25.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.924+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c881544b0 con 0x7f4c88151fc0 2024-03-16T19:57:25.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.928+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2e940f0890 con 0x7f2ea415d570 2024-03-16T19:57:25.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.928+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80012620 con 0x7f4c88151fc0 2024-03-16T19:57:25.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.928+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8806c5c0 con 0x7f4c88151fc0 2024-03-16T19:57:25.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.928+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8806c7b0 con 0x7f4c88151fc0 2024-03-16T19:57:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:25.944+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2e940ec430 con 0x7f2ea415d570 2024-03-16T19:57:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.020+0000 7f89a6ffd640 1 -- 172.21.15.32:0/1458256373 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f89b8070b20 con 0x7f89b81e5aa0 2024-03-16T19:57:25.993 INFO:teuthology.orchestra.run.smithi032.stdout:64424509461 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 msgr2=0x7f8998069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 0x7f8998069350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f89a8007a80 tx=0x7f89a8001560 comp rx=0 tx=0).stop 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 msgr2=0x7f89b81f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89b00987a0 tx=0x7f89a0005800 comp rx=0 tx=0).stop 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 shutdown_connections 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8998066e90 0x7f8998069350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89b81e5aa0 0x7f89b81f0300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f89b814c980 0x7f89b81e5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 --2- 172.21.15.32:0/1458256373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b8147980 0x7f89b814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 >> 172.21.15.32:0/1458256373 conn(0x7f89b80cc7a0 msgr2=0x7f89b8074080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 shutdown_connections 2024-03-16T19:57:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.024+0000 7f89bcb45640 1 -- 172.21.15.32:0/1458256373 wait complete. 2024-03-16T19:57:26.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.032+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fedd80c3830 con 0x7fedd815e910 2024-03-16T19:57:26.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509461 for osd.0 2024-03-16T19:57:26.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.104+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2ea4068000 con 0x7f2ea415d570 2024-03-16T19:57:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.112+0000 7f2e99ffb640 1 -- 172.21.15.32:0/3055621350 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e940efaf0 con 0x7f2ea415d570 2024-03-16T19:57:26.084 INFO:teuthology.orchestra.run.smithi032.stdout:68719476759 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 msgr2=0x7f2e88069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 0x7f2e88069330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0007fd0 tx=0x7f2ea0009040 comp rx=0 tx=0).stop 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 msgr2=0x7f2ea415c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e9401a0e0 tx=0x7f2e94019250 comp rx=0 tx=0).stop 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 shutdown_connections 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e88066e70 0x7f2e88069330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ea415e070 0x7f2ea4148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea415d570 0x7f2ea415c0f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 --2- 172.21.15.32:0/3055621350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ea4154560 0x7f2ea414bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 >> 172.21.15.32:0/3055621350 conn(0x7f2ea406f3a0 msgr2=0x7f2ea40c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 shutdown_connections 2024-03-16T19:57:26.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.116+0000 7f2eab5a4640 1 -- 172.21.15.32:0/3055621350 wait complete. 2024-03-16T19:57:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.128+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1c98056170 con 0x7f1ca80725b0 2024-03-16T19:57:26.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476760 got 68719476759 for osd.6 2024-03-16T19:57:26.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.220+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1c98055840 con 0x7f1ca80725b0 2024-03-16T19:57:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4c80007070 con 0x7f4c88151fc0 2024-03-16T19:57:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7f4c7e7fc640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 0x7f4c6c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7f4c8d893640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 0x7f4c6c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7f4c8d893640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 0x7f4c6c069300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009650 tx=0x7f4c84006bb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:26.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f1332640 1 -- 172.21.15.32:0/517950401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40af400 con 0x7fa1ec0c87b0 2024-03-16T19:57:26.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f1332640 1 -- 172.21.15.32:0/517950401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40ab050 con 0x7fa1ec0c87b0 2024-03-16T19:57:26.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f1332640 1 -- 172.21.15.32:0/517950401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40af780 con 0x7fa1ec0c87b0 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 -- 172.21.15.32:0/517950401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c87b0 msgr2=0x7fa1ec160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 --2- 172.21.15.32:0/517950401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec160960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009e50 tx=0x7fa1e401f200 comp rx=0 tx=0).stop 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 -- 172.21.15.32:0/517950401 shutdown_connections 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 --2- 172.21.15.32:0/517950401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec160960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 --2- 172.21.15.32:0/517950401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec0c68e0 0x7fa1ec0c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 -- 172.21.15.32:0/517950401 >> 172.21.15.32:0/517950401 conn(0x7fa1ec06afc0 msgr2=0x7fa1ec06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 -- 172.21.15.32:0/517950401 shutdown_connections 2024-03-16T19:57:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.416+0000 7fa1f4598640 1 -- 172.21.15.32:0/517950401 wait complete. 2024-03-16T19:57:26.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 Processor -- start 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 -- start start 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c68e0 0x7fa1ec1f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1f0000 con 0x7fa1ec0c68e0 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c48a0 con 0x7fa1ec0c87b0 2024-03-16T19:57:26.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f4598640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1ef9f0 con 0x7fa1ec14caa0 2024-03-16T19:57:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f2b35640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f2b35640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44106/0 (socket says 172.21.15.32:44106) 2024-03-16T19:57:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f2b35640 1 -- 172.21.15.32:0/2303387612 learned_addr learned my addr 172.21.15.32:0/2303387612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7fa1f1b33640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.420+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1c8c005820 con 0x7f1ca80725b0 2024-03-16T19:57:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.428+0000 7fa1f2b35640 1 -- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1ec0c87b0 msgr2=0x7fa1ec14e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.428+0000 7fa1f2b35640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec14e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.428+0000 7fa1f2b35640 1 -- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c68e0 msgr2=0x7fa1ec1f0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.428+0000 7fa1f2b35640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c68e0 0x7fa1ec1f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.428+0000 7fa1f2b35640 1 -- 172.21.15.32:0/2303387612 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4009b30 con 0x7fa1ec14caa0 2024-03-16T19:57:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.432+0000 7fa1f2b35640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1e401d8b0 tx=0x7fa1e401bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:26.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.432+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4c800560a0 con 0x7f4c88151fc0 2024-03-16T19:57:26.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.432+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4c80051c40 con 0x7f4c88151fc0 2024-03-16T19:57:26.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.440+0000 7fedbf7fe640 1 -- 172.21.15.32:0/1510654823 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedd80c3830 con 0x7fedd815e910 2024-03-16T19:57:26.412 INFO:teuthology.orchestra.run.smithi032.stdout:68719476758 2024-03-16T19:57:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 msgr2=0x7fedc4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 0x7fedc4069350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009690 tx=0x7fedc8006c00 comp rx=0 tx=0).stop 2024-03-16T19:57:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 msgr2=0x7fedd81e85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fedd000ca20 tx=0x7fedd000cee0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 shutdown_connections 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fedc4066e90 0x7fedc4069350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd815e910 0x7fedd81e85c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedd8158700 0x7fedd8160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 --2- 172.21.15.32:0/1510654823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedd8073110 0x7fedd81f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 >> 172.21.15.32:0/1510654823 conn(0x7fedd80cc7a0 msgr2=0x7fedd80c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 shutdown_connections 2024-03-16T19:57:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.444+0000 7feddd63e640 1 -- 172.21.15.32:0/1510654823 wait complete. 2024-03-16T19:57:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7f1c9f7fe640 1 -- 172.21.15.32:0/2720252724 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c98014090 con 0x7f1ca80725b0 2024-03-16T19:57:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4009510 con 0x7fa1ec14caa0 2024-03-16T19:57:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40b8400 con 0x7fa1ec14caa0 2024-03-16T19:57:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec1508d0 con 0x7fa1ec14caa0 2024-03-16T19:57:26.429 INFO:teuthology.orchestra.run.smithi032.stdout:64424509461 2024-03-16T19:57:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b8790 con 0x7fa1ec14caa0 2024-03-16T19:57:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.456+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec1efc90 con 0x7fa1ec14caa0 2024-03-16T19:57:26.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec06a340 con 0x7fa1ec14caa0 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 msgr2=0x7f1c88069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 0x7f1c88069350 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1c900b2000 tx=0x7f1c90004740 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 msgr2=0x7f1ca80cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b580 tx=0x7f1c9800ba40 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 shutdown_connections 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1c88066e90 0x7f1c88069350 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ca815a410 0x7f1ca8158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ca8073180 0x7f1ca81580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 --2- 172.21.15.32:0/2720252724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ca80725b0 0x7f1ca80cb970 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 >> 172.21.15.32:0/2720252724 conn(0x7f1ca80cc7a0 msgr2=0x7f1ca80c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 shutdown_connections 2024-03-16T19:57:26.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.460+0000 7f1cad7ad640 1 -- 172.21.15.32:0/2720252724 wait complete. 2024-03-16T19:57:26.435 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476759 got 68719476758 for osd.5 2024-03-16T19:57:26.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509461 for osd.2 2024-03-16T19:57:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.688+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4c88068000 con 0x7f4c88151fc0 2024-03-16T19:57:26.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.696+0000 7f4c7e7fc640 1 -- 172.21.15.32:0/1496667451 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c80017020 con 0x7f4c88151fc0 2024-03-16T19:57:26.667 INFO:teuthology.orchestra.run.smithi032.stdout:64424509461 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 msgr2=0x7f4c6c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 0x7f4c6c069300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009650 tx=0x7f4c84006bb0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 msgr2=0x7f4c8806cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000caa0 tx=0x7f4c8000cf60 comp rx=0 tx=0).stop 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 shutdown_connections 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4c6c066e40 0x7f4c6c069300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c88151fc0 0x7f4c8806cae0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c880c8d00 0x7f4c880c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 --2- 172.21.15.32:0/1496667451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c880705b0 0x7f4c88154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 >> 172.21.15.32:0/1496667451 conn(0x7f4c88066ea0 msgr2=0x7f4c88067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 shutdown_connections 2024-03-16T19:57:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.700+0000 7f4c8faf7640 1 -- 172.21.15.32:0/1496667451 wait complete. 2024-03-16T19:57:26.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509460 got 64424509461 for osd.3 2024-03-16T19:57:26.686 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:26.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:57:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.788+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa1e40b6040 con 0x7fa1ec14caa0 2024-03-16T19:57:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.792+0000 7fa1e2ffd640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 0x7fa1d0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.792+0000 7fa1f2334640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 0x7fa1d0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.792+0000 7fa1f2334640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 0x7fa1d0069380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8007fd0 tx=0x7fa1e80079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:26.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.796+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa1e40f4b00 con 0x7fa1ec14caa0 2024-03-16T19:57:26.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.816+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa1e40f06a0 con 0x7fa1ec14caa0 2024-03-16T19:57:26.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 Processor -- start 2024-03-16T19:57:26.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 -- start start 2024-03-16T19:57:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85d00c5f40 0x7f85d0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d0158560 0x7f85d01519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d006fd00 con 0x7f85d00c6320 2024-03-16T19:57:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d0070b00 con 0x7f85d0158560 2024-03-16T19:57:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d8477640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d006ff30 con 0x7f85d00c5f40 2024-03-16T19:57:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.848+0000 7f85d6a14640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d0158560 0x7f85d01519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d0158560 0x7f85d01519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59862/0 (socket says 172.21.15.32:59862) 2024-03-16T19:57:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 -- 172.21.15.32:0/2123618222 learned_addr learned my addr 172.21.15.32:0/2123618222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6213640 1 --2- 172.21.15.32:0/2123618222 >> v2:172.21.15.59:3300/0 conn(0x7f85d00c5f40 0x7f85d0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 -- 172.21.15.32:0/2123618222 >> v2:172.21.15.59:3300/0 conn(0x7f85d00c5f40 msgr2=0x7f85d0072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 --2- 172.21.15.32:0/2123618222 >> v2:172.21.15.59:3300/0 conn(0x7f85d00c5f40 0x7f85d0072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 -- 172.21.15.32:0/2123618222 >> v1:172.21.15.59:6789/0 conn(0x7f85d00c6320 legacy=0x7f85d0157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.852+0000 7f85d6a14640 1 -- 172.21.15.32:0/2123618222 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00701d0 con 0x7f85d0158560 2024-03-16T19:57:26.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.856+0000 7f85d6a14640 1 --2- 172.21.15.32:0/2123618222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d0158560 0x7f85d01519f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f85d00c6b10 tx=0x7f85c801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47599147753ce4f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:26.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d5211640 1 -- 172.21.15.32:0/2123618222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80a56d0 con 0x7f85d0158560 2024-03-16T19:57:26.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d5211640 1 -- 172.21.15.32:0/2123618222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c80a5830 con 0x7f85d0158560 2024-03-16T19:57:26.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d5211640 1 -- 172.21.15.32:0/2123618222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80a5bc0 con 0x7f85d0158560 2024-03-16T19:57:26.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 -- 172.21.15.32:0/2123618222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d0158560 msgr2=0x7f85d01519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 --2- 172.21.15.32:0/2123618222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d0158560 0x7f85d01519f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f85d00c6b10 tx=0x7f85c801c920 comp rx=0 tx=0).stop 2024-03-16T19:57:26.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 -- 172.21.15.32:0/2123618222 shutdown_connections 2024-03-16T19:57:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 --2- 172.21.15.32:0/2123618222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d0158560 0x7f85d01519f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 --2- 172.21.15.32:0/2123618222 >> v2:172.21.15.59:3300/0 conn(0x7f85d00c5f40 0x7f85d0072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.860+0000 7f85d8477640 1 -- 172.21.15.32:0/2123618222 >> 172.21.15.32:0/2123618222 conn(0x7f85d00cc7a0 msgr2=0x7f85d00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2123618222 shutdown_connections 2024-03-16T19:57:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2123618222 wait complete. 2024-03-16T19:57:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 Processor -- start 2024-03-16T19:57:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- start start 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85d00c6320 0x7f85d01453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85d0158560 0x7f85d00c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d01478a0 con 0x7f85d00c5f40 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0146ec0 con 0x7f85d00c6320 2024-03-16T19:57:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d8477640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0147180 con 0x7f85d0158560 2024-03-16T19:57:26.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d6213640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d6213640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59870/0 (socket says 172.21.15.32:59870) 2024-03-16T19:57:26.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d6213640 1 -- 172.21.15.32:0/2786360856 learned_addr learned my addr 172.21.15.32:0/2786360856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:26.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.864+0000 7f85d5a12640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85d00c6320 0x7f85d01453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.868+0000 7f85d6213640 1 -- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85d0158560 msgr2=0x7f85d00c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.868+0000 7f85d6213640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85d0158560 0x7f85d00c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.868+0000 7f85d6213640 1 -- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85d00c6320 msgr2=0x7f85d01453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:26.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.868+0000 7f85d6213640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85d00c6320 0x7f85d01453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:26.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.868+0000 7f85d6213640 1 -- 172.21.15.32:0/2786360856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009b30 con 0x7f85d00c5f40 2024-03-16T19:57:26.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.872+0000 7f85d6213640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d5f0 tx=0x7f85cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:26.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.876+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0125f0 con 0x7f85d00c5f40 2024-03-16T19:57:26.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.876+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc012750 con 0x7f85d00c5f40 2024-03-16T19:57:26.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.876+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc012b00 con 0x7f85d00c5f40 2024-03-16T19:57:26.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.876+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d0147460 con 0x7f85d00c5f40 2024-03-16T19:57:26.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.876+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d00c40e0 con 0x7f85d00c5f40 2024-03-16T19:57:26.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.880+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d00c3c60 con 0x7f85d00c5f40 2024-03-16T19:57:26.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:57:26.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.996+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85cc012c60 con 0x7f85d00c5f40 2024-03-16T19:57:26.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.996+0000 7f85c6ffd640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 0x7f85b4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:26.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.996+0000 7f85d5a12640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 0x7f85b4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:26.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:26.996+0000 7f85d5a12640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 0x7f85b4069350 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85d01480a0 tx=0x7f85c0009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:57:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.052+0000 7f6fc6062640 1 Processor -- start 2024-03-16T19:57:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.052+0000 7f6fc6062640 1 -- start start 2024-03-16T19:57:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fc6062640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6fc0150900 0x7f6fc015a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fc6062640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00c7800 con 0x7f6fc00c6890 2024-03-16T19:57:27.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fc6062640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc0072c80 con 0x7f6fc00b1da0 2024-03-16T19:57:27.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fc6062640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc0073490 con 0x7f6fc0150900 2024-03-16T19:57:27.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fbeffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6fc00b1da0 0x7f6fc0071f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52658/0 (socket says 172.21.15.32:52658) 2024-03-16T19:57:27.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.056+0000 7f6fbeffd640 1 -- 172.21.15.32:0/2165263008 learned_addr learned my addr 172.21.15.32:0/2165263008 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.064+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077620493 0 0) 0x7f6fc0072c80 con 0x7f6fc00b1da0 2024-03-16T19:57:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.064+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa8003450 con 0x7f6fc00b1da0 2024-03-16T19:57:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014067342 0 0) 0x7f6fa8003450 con 0x7f6fc00b1da0 2024-03-16T19:57:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 >> v2:172.21.15.32:3300/0 conn(0x7f6fc0150900 msgr2=0x7f6fc015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 --2- 172.21.15.32:0/2165263008 >> v2:172.21.15.32:3300/0 conn(0x7f6fc0150900 0x7f6fc015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 >> v1:172.21.15.32:6789/0 conn(0x7f6fc00c6890 legacy=0x7f6fc0160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00c3020 con 0x7f6fc00b1da0 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f6fb40016d0 con 0x7f6fc00b1da0 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 >> v1:172.21.15.59:6789/0 conn(0x7f6fc00b1da0 legacy=0x7f6fc0071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fa8003ac0 0x7f6fa8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 0x7f6fa800ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa800b660 0x7f6fa800db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0073490 con 0x7f6fa800b660 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa800e400 con 0x7f6fa8003ac0 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa800e5c0 con 0x7f6fa80089f0 2024-03-16T19:57:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.076+0000 7f6fbeffd640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 0x7f6fa800ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbf7fe640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fa8003ac0 0x7f6fa8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbeffd640 1 -- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fa8003ac0 msgr2=0x7f6fa8003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbeffd640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fa8003ac0 0x7f6fa8003ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbeffd640 1 -- 172.21.15.32:0/2165263008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa800b660 msgr2=0x7f6fa800db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbeffd640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa800b660 0x7f6fa800db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbffff640 1 -- 172.21.15.32:0/2165263008 reap_dead start 2024-03-16T19:57:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.084+0000 7f6fbeffd640 1 -- 172.21.15.32:0/2165263008 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8003450 con 0x7f6fa80089f0 2024-03-16T19:57:27.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.088+0000 7f6fbeffd640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 0x7f6fa800ae60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400c110 tx=0x7f6fb401cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e29c1bd16484d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.096+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4005ae0 con 0x7f6fa80089f0 2024-03-16T19:57:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.096+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb4005c60 con 0x7f6fa80089f0 2024-03-16T19:57:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.096+0000 7f6fbe7fc640 1 -- 172.21.15.32:0/2165263008 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40a5ac0 con 0x7f6fa80089f0 2024-03-16T19:57:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.096+0000 7f6fc6062640 1 -- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 msgr2=0x7f6fa800ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 0x7f6fa800ae60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400c110 tx=0x7f6fb401cb80 comp rx=0 tx=0).stop 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- 172.21.15.32:0/2165263008 shutdown_connections 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2165263008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fa80089f0 0x7f6fa800ae60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- 172.21.15.32:0/2165263008 >> 172.21.15.32:0/2165263008 conn(0x7f6fc006f3a0 msgr2=0x7f6fc006f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- 172.21.15.32:0/2165263008 shutdown_connections 2024-03-16T19:57:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- 172.21.15.32:0/2165263008 wait complete. 2024-03-16T19:57:27.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 Processor -- start 2024-03-16T19:57:27.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- start start 2024-03-16T19:57:27.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa80089f0 0x7f6fc0153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 0x7f6fc014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc006c8e0 con 0x7f6fa80089f0 2024-03-16T19:57:27.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc01ecae0 con 0x7f6fc0150d70 2024-03-16T19:57:27.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.100+0000 7f6fc6062640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0153090 con 0x7f6fc014bf50 2024-03-16T19:57:27.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f6fbeffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 0x7f6fc014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f6fbffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f6fbeffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 0x7f6fc014a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44130/0 (socket says 172.21.15.32:44130) 2024-03-16T19:57:27.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f6fbffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:43970/0 (socket says 172.21.15.32:43970) 2024-03-16T19:57:27.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f6fbeffd640 1 -- 172.21.15.32:0/2712690349 learned_addr learned my addr 172.21.15.32:0/2712690349 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.104+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f85cc056a20 con 0x7f85d00c5f40 2024-03-16T19:57:27.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa1ec0c6370 con 0x7fa1ec14caa0 2024-03-16T19:57:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7f6fbffff640 1 -- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 msgr2=0x7f6fc014a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7f6fbffff640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 0x7f6fc014a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7f6fbffff640 1 -- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa80089f0 msgr2=0x7f6fc0153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7f6fbffff640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa80089f0 0x7f6fc0153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.108+0000 7f6fbffff640 1 -- 172.21.15.32:0/2712690349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc01ec580 con 0x7f6fc0150d70 2024-03-16T19:57:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.112+0000 7f6fbffff640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0071610 tx=0x7f6fb800c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8004d30 con 0x7f6fc0150d70 2024-03-16T19:57:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8011620 con 0x7f6fc0150d70 2024-03-16T19:57:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8011a10 con 0x7f6fc0150d70 2024-03-16T19:57:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc01ec800 con 0x7f6fc0150d70 2024-03-16T19:57:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc0152d20 con 0x7f6fc0150d70 2024-03-16T19:57:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.124+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc0152ee0 con 0x7f6fc0150d70 2024-03-16T19:57:27.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1e2ffd640 1 -- 172.21.15.32:0/2303387612 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1e40b4070 con 0x7fa1ec14caa0 2024-03-16T19:57:27.099 INFO:teuthology.orchestra.run.smithi032.stdout:68719476759 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 msgr2=0x7fa1d0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 0x7fa1d0069380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8007fd0 tx=0x7fa1e80079a0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 msgr2=0x7fa1ec1f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1e401d8b0 tx=0x7fa1e401bfa0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 shutdown_connections 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa1d0066ec0 0x7fa1d0069380 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1ec14caa0 0x7fa1ec1f3830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1ec0c87b0 0x7fa1ec14e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 --2- 172.21.15.32:0/2303387612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec0c68e0 0x7fa1ec1f0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 >> 172.21.15.32:0/2303387612 conn(0x7fa1ec06afc0 msgr2=0x7fa1ec0c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 shutdown_connections 2024-03-16T19:57:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.128+0000 7fa1f4598640 1 -- 172.21.15.32:0/2303387612 wait complete. 2024-03-16T19:57:27.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:57:27.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476758 got 68719476759 for osd.7 2024-03-16T19:57:27.124 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 Processor -- start 2024-03-16T19:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 -- start start 2024-03-16T19:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f052c151e50 0x7f052c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c15bdf0 con 0x7f052c0c97b0 2024-03-16T19:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f052c158d30 con 0x7f052c151e50 2024-03-16T19:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f0531817640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c152230 con 0x7f052c06f0a0 2024-03-16T19:57:27.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f052a7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f052a7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44144/0 (socket says 172.21.15.32:44144) 2024-03-16T19:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.184+0000 7f052a7fc640 1 -- 172.21.15.32:0/1647992599 learned_addr learned my addr 172.21.15.32:0/1647992599 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.188+0000 7f052a7fc640 1 -- 172.21.15.32:0/1647992599 >> v2:172.21.15.59:3300/0 conn(0x7f052c151e50 msgr2=0x7f052c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.188+0000 7f052a7fc640 1 --2- 172.21.15.32:0/1647992599 >> v2:172.21.15.59:3300/0 conn(0x7f052c151e50 0x7f052c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.188+0000 7f052a7fc640 1 -- 172.21.15.32:0/1647992599 >> v1:172.21.15.59:6789/0 conn(0x7f052c0c97b0 legacy=0x7f052c157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.188+0000 7f052a7fc640 1 -- 172.21.15.32:0/1647992599 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c15c280 con 0x7f052c06f0a0 2024-03-16T19:57:27.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.196+0000 7f052a7fc640 1 --2- 172.21.15.32:0/1647992599 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0520009870 tx=0x7f052001ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2abd9964e04db556 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.200+0000 7f0529ffb640 1 -- 172.21.15.32:0/1647992599 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200a89d0 con 0x7f052c06f0a0 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.200+0000 7f0529ffb640 1 -- 172.21.15.32:0/1647992599 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05200a8b30 con 0x7f052c06f0a0 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.200+0000 7f0529ffb640 1 -- 172.21.15.32:0/1647992599 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200b0440 con 0x7f052c06f0a0 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.200+0000 7f0531817640 1 -- 172.21.15.32:0/1647992599 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 msgr2=0x7f052c0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.200+0000 7f0531817640 1 --2- 172.21.15.32:0/1647992599 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0520009870 tx=0x7f052001ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- 172.21.15.32:0/1647992599 shutdown_connections 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 --2- 172.21.15.32:0/1647992599 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c06f0a0 0x7f052c0c81e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 --2- 172.21.15.32:0/1647992599 >> v2:172.21.15.59:3300/0 conn(0x7f052c151e50 0x7f052c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- 172.21.15.32:0/1647992599 >> 172.21.15.32:0/1647992599 conn(0x7f052c070f90 msgr2=0x7f052c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- 172.21.15.32:0/1647992599 shutdown_connections 2024-03-16T19:57:27.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- 172.21.15.32:0/1647992599 wait complete. 2024-03-16T19:57:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 Processor -- start 2024-03-16T19:57:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- start start 2024-03-16T19:57:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f052c06f0a0 0x7f052c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c0c97b0 0x7f052c0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0ce5a0 con 0x7f052c06f0a0 2024-03-16T19:57:27.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0b3120 con 0x7f052c0c97b0 2024-03-16T19:57:27.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f0531817640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0c5690 con 0x7f052c151e50 2024-03-16T19:57:27.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.204+0000 7f052b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.208+0000 7f052b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44158/0 (socket says 172.21.15.32:44158) 2024-03-16T19:57:27.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.208+0000 7f052b7fe640 1 -- 172.21.15.32:0/3024398633 learned_addr learned my addr 172.21.15.32:0/3024398633 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.212+0000 7f052b7fe640 1 -- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c0c97b0 msgr2=0x7f052c0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.212+0000 7f052b7fe640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c0c97b0 0x7f052c0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.212+0000 7f052b7fe640 1 -- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f052c06f0a0 msgr2=0x7f052c06bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.212+0000 7f052b7fe640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f052c06f0a0 0x7f052c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.212+0000 7f052b7fe640 1 -- 172.21.15.32:0/3024398633 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200094d0 con 0x7f052c151e50 2024-03-16T19:57:27.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.216+0000 7f052b7fe640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f052400c780 tx=0x7f052400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524003d90 con 0x7f052c151e50 2024-03-16T19:57:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052400ce30 con 0x7f052c151e50 2024-03-16T19:57:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524005290 con 0x7f052c151e50 2024-03-16T19:57:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c0c5970 con 0x7f052c151e50 2024-03-16T19:57:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c0c5f30 con 0x7f052c151e50 2024-03-16T19:57:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.224+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c0c6150 con 0x7f052c151e50 2024-03-16T19:57:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.300+0000 7fc4d3ae3640 1 Processor -- start 2024-03-16T19:57:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.300+0000 7fc4d3ae3640 1 -- start start 2024-03-16T19:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.304+0000 7fc4d3ae3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4cc0b1a90 0x7fc4cc15b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.304+0000 7fc4d3ae3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.304+0000 7fc4d3ae3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc072c90 con 0x7fc4cc0c54b0 2024-03-16T19:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.304+0000 7fc4d3ae3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4cc0c5890 con 0x7fc4cc0c7560 2024-03-16T19:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.304+0000 7fc4d3ae3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4cc072f00 con 0x7fc4cc0b1a90 2024-03-16T19:57:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.328+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f85cc0525c0 con 0x7f85d00c5f40 2024-03-16T19:57:27.327 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-16T19:57:27.312+0000 f9ea640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T19:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.388+0000 7fc4d2080640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.388+0000 7fc4d2080640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59922/0 (socket says 172.21.15.32:59922) 2024-03-16T19:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.388+0000 7fc4d2080640 1 -- 172.21.15.32:0/1665191467 learned_addr learned my addr 172.21.15.32:0/1665191467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.396+0000 7fc4d2080640 1 -- 172.21.15.32:0/1665191467 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc0b1a90 msgr2=0x7fc4cc15b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.396+0000 7fc4d2080640 1 --2- 172.21.15.32:0/1665191467 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc0b1a90 0x7fc4cc15b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.396+0000 7fc4d2080640 1 -- 172.21.15.32:0/1665191467 >> v1:172.21.15.59:6789/0 conn(0x7fc4cc0c54b0 legacy=0x7fc4cc0c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.396+0000 7fc4d2080640 1 -- 172.21.15.32:0/1665191467 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0731e0 con 0x7fc4cc0c7560 2024-03-16T19:57:27.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.400+0000 7fc4d2080640 1 --2- 172.21.15.32:0/1665191467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0c35a0 tx=0x7fc4c401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c25ab91c9d4cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d087d640 1 -- 172.21.15.32:0/1665191467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40af400 con 0x7fc4cc0c7560 2024-03-16T19:57:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d087d640 1 -- 172.21.15.32:0/1665191467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c40ab050 con 0x7fc4cc0c7560 2024-03-16T19:57:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d087d640 1 -- 172.21.15.32:0/1665191467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40af780 con 0x7fc4cc0c7560 2024-03-16T19:57:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/1665191467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 msgr2=0x7fc4cc15b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/1665191467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0c35a0 tx=0x7fc4c401f1a0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/1665191467 shutdown_connections 2024-03-16T19:57:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/1665191467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc15b8f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/1665191467 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc0b1a90 0x7fc4cc15b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/1665191467 >> 172.21.15.32:0/1665191467 conn(0x7fc4cc06f3a0 msgr2=0x7fc4cc06f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/1665191467 shutdown_connections 2024-03-16T19:57:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.408+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/1665191467 wait complete. 2024-03-16T19:57:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 Processor -- start 2024-03-16T19:57:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 -- start start 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc0b1a90 0x7fc4cc1e3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0c54b0 0x7fc4cc154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc155980 con 0x7fc4cc0c7560 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc1de5c0 con 0x7fc4cc0c54b0 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d3ae3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc1e3740 con 0x7fc4cc0b1a90 2024-03-16T19:57:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d2080640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d2080640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59932/0 (socket says 172.21.15.32:59932) 2024-03-16T19:57:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.412+0000 7fc4d2080640 1 -- 172.21.15.32:0/380461150 learned_addr learned my addr 172.21.15.32:0/380461150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.420+0000 7fc4d2080640 1 -- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc0b1a90 msgr2=0x7fc4cc1e3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.420+0000 7fc4d2080640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc0b1a90 0x7fc4cc1e3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.420+0000 7fc4d2080640 1 -- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0c54b0 msgr2=0x7fc4cc154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.420+0000 7fc4d2080640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0c54b0 0x7fc4cc154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.420+0000 7fc4d2080640 1 -- 172.21.15.32:0/380461150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4009b30 con 0x7fc4cc0c7560 2024-03-16T19:57:27.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.424+0000 7fc4d2080640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40b2040 tx=0x7fc4c401bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.428+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40afde0 con 0x7fc4cc0c7560 2024-03-16T19:57:27.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.428+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc1de2e0 con 0x7fc4cc0c7560 2024-03-16T19:57:27.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.428+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc1556a0 con 0x7fc4cc0c7560 2024-03-16T19:57:27.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.428+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc068000 con 0x7fc4cc0c7560 2024-03-16T19:57:27.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:57:27.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:57:27.472 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-16T19:57:27.316+0000 f9ea640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-16T19:57:27.472 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-16T19:57:27.316+0000 f9ea640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-16T19:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.552+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0524006b20 con 0x7f052c151e50 2024-03-16T19:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f0513fff640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 0x7f0518069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.552+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6fb8011b70 con 0x7f6fc0150d70 2024-03-16T19:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f6fa7fff640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 0x7f6fac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f6fbf7fe640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 0x7f6fac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f052affd640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 0x7f0518069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f6fbf7fe640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 0x7f6fac069350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000a7b0 tx=0x7f6fb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.556+0000 7f052affd640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 0x7f0518069300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f051c009690 tx=0x7f051c006c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.560+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6fb8011e20 con 0x7f6fc0150d70 2024-03-16T19:57:27.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.560+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0524055630 con 0x7f052c151e50 2024-03-16T19:57:27.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.572+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6fb8054f40 con 0x7f6fc0150d70 2024-03-16T19:57:27.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.572+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f85d00c3c60 con 0x7f85d00c5f40 2024-03-16T19:57:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.580+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f05240511d0 con 0x7f052c151e50 2024-03-16T19:57:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.616+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4009510 con 0x7fc4cc0c7560 2024-03-16T19:57:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.616+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40c6660 con 0x7fc4cc0c7560 2024-03-16T19:57:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.616+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc4c40b6040 con 0x7fc4cc0c7560 2024-03-16T19:57:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.620+0000 7fc4c27fc640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 0x7fc4b00692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.620+0000 7fc4d187f640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 0x7fc4b00692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.620+0000 7fc4d187f640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 0x7fc4b00692e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007fd0 tx=0x7fc4c80079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.624+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc4c40f48c0 con 0x7fc4cc0c7560 2024-03-16T19:57:27.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.652+0000 7f9968f3b640 1 Processor -- start 2024-03-16T19:57:27.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.652+0000 7f9968f3b640 1 -- start start 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9968f3b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 0x7f99640c7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9968f3b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99640c2900 0x7f996406cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9968f3b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99640c7a60 con 0x7f99640c2d20 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9968f3b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99640c7cf0 con 0x7f99640c2900 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9968f3b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996415b250 con 0x7f9964151c30 2024-03-16T19:57:27.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9963fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f99640c2d20 0x7f9964157c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52678/0 (socket says 172.21.15.32:52678) 2024-03-16T19:57:27.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f9963fff640 1 -- 172.21.15.32:0/3208477562 learned_addr learned my addr 172.21.15.32:0/3208477562 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:27.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.656+0000 7f99637fe640 1 --2- 172.21.15.32:0/3208477562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 0x7f99640c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99627fc640 1 -- 172.21.15.32:0/3208477562 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919159590 0 0) 0x7f99640c7a60 con 0x7f99640c2d20 2024-03-16T19:57:27.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99627fc640 1 -- 172.21.15.32:0/3208477562 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c003530 con 0x7f99640c2d20 2024-03-16T19:57:27.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99637fe640 1 -- 172.21.15.32:0/3208477562 >> v2:172.21.15.32:3300/0 conn(0x7f99640c2900 msgr2=0x7f996406cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99637fe640 1 --2- 172.21.15.32:0/3208477562 >> v2:172.21.15.32:3300/0 conn(0x7f99640c2900 0x7f996406cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99637fe640 1 -- 172.21.15.32:0/3208477562 >> v1:172.21.15.59:6789/0 conn(0x7f99640c2d20 legacy=0x7f9964157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.660+0000 7f99637fe640 1 -- 172.21.15.32:0/3208477562 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996415b530 con 0x7f9964151c30 2024-03-16T19:57:27.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.664+0000 7f99637fe640 1 --2- 172.21.15.32:0/3208477562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 0x7f99640c7520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99580097c0 tx=0x7f995801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b137cdc1486e4168 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f99627fc640 1 -- 172.21.15.32:0/3208477562 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580a8990 con 0x7f9964151c30 2024-03-16T19:57:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f99627fc640 1 -- 172.21.15.32:0/3208477562 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99580a8af0 con 0x7f9964151c30 2024-03-16T19:57:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f99627fc640 1 -- 172.21.15.32:0/3208477562 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580a8e80 con 0x7f9964151c30 2024-03-16T19:57:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- 172.21.15.32:0/3208477562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 msgr2=0x7f99640c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3208477562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 0x7f99640c7520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99580097c0 tx=0x7f995801ecb0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- 172.21.15.32:0/3208477562 shutdown_connections 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3208477562 >> v2:172.21.15.32:3300/0 conn(0x7f99640c2900 0x7f996406cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3208477562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9964151c30 0x7f99640c7520 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- 172.21.15.32:0/3208477562 >> 172.21.15.32:0/3208477562 conn(0x7f996406b190 msgr2=0x7f996406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- 172.21.15.32:0/3208477562 shutdown_connections 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- 172.21.15.32:0/3208477562 wait complete. 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 Processor -- start 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.672+0000 7f9968f3b640 1 -- start start 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99640c2900 0x7f99640722a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 0x7f996414fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 0x7f996414ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964156580 con 0x7f99640c2900 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964157410 con 0x7f9964151c30 2024-03-16T19:57:27.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9968f3b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99640c8520 con 0x7f99640c2d20 2024-03-16T19:57:27.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9962ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 0x7f996414fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9963fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 0x7f996414ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9962ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 0x7f996414fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44198/0 (socket says 172.21.15.32:44198) 2024-03-16T19:57:27.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.676+0000 7f9962ffd640 1 -- 172.21.15.32:0/3176845693 learned_addr learned my addr 172.21.15.32:0/3176845693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.680+0000 7f9963fff640 1 -- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 msgr2=0x7f996414fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.680+0000 7f9963fff640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 0x7f996414fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.680+0000 7f9963fff640 1 -- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99640c2900 msgr2=0x7f99640722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.680+0000 7f9963fff640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99640c2900 0x7f99640722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.680+0000 7f9963fff640 1 -- 172.21.15.32:0/3176845693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958009420 con 0x7f9964151c30 2024-03-16T19:57:27.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.684+0000 7f47a87ba640 1 Processor -- start 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.684+0000 7f47a87ba640 1 -- start start 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a87ba640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f47a0158830 0x7f47a006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a87ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a87ba640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a0158c10 con 0x7f47a0071650 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a87ba640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00c2b30 con 0x7f47a0158830 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a87ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00c3870 con 0x7f47a00c8f30 2024-03-16T19:57:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f9963fff640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 0x7f996414ff50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f995400c340 tx=0x7f995400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a6d57640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a6d57640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44200/0 (socket says 172.21.15.32:44200) 2024-03-16T19:57:27.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.688+0000 7f47a6d57640 1 -- 172.21.15.32:0/1648018874 learned_addr learned my addr 172.21.15.32:0/1648018874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.692+0000 7f47a6d57640 1 -- 172.21.15.32:0/1648018874 >> v2:172.21.15.59:3300/0 conn(0x7f47a0158830 msgr2=0x7f47a006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.692+0000 7f47a6d57640 1 --2- 172.21.15.32:0/1648018874 >> v2:172.21.15.59:3300/0 conn(0x7f47a0158830 0x7f47a006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.692+0000 7f47a6d57640 1 -- 172.21.15.32:0/1648018874 >> v1:172.21.15.59:6789/0 conn(0x7f47a0071650 legacy=0x7f47a00c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:27.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.692+0000 7f47a6d57640 1 -- 172.21.15.32:0/1648018874 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a015f3d0 con 0x7f47a00c8f30 2024-03-16T19:57:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954005cf0 con 0x7f9964151c30 2024-03-16T19:57:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954005e50 con 0x7f9964151c30 2024-03-16T19:57:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540072f0 con 0x7f9964151c30 2024-03-16T19:57:27.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f47a6d57640 1 --2- 172.21.15.32:0/1648018874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47a00cc990 tx=0x7f479801c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e555748d056ed0d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9964150d80 con 0x7f9964151c30 2024-03-16T19:57:27.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9964151320 con 0x7f9964151c30 2024-03-16T19:57:27.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.700+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99440052a0 con 0x7f9964151c30 2024-03-16T19:57:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a5554640 1 -- 172.21.15.32:0/1648018874 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980a5820 con 0x7f47a00c8f30 2024-03-16T19:57:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a5554640 1 -- 172.21.15.32:0/1648018874 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980a5980 con 0x7f47a00c8f30 2024-03-16T19:57:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a5554640 1 -- 172.21.15.32:0/1648018874 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980a5d10 con 0x7f47a00c8f30 2024-03-16T19:57:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 -- 172.21.15.32:0/1648018874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 msgr2=0x7f47a0161da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 --2- 172.21.15.32:0/1648018874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47a00cc990 tx=0x7f479801c9a0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 -- 172.21.15.32:0/1648018874 shutdown_connections 2024-03-16T19:57:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 --2- 172.21.15.32:0/1648018874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a00c8f30 0x7f47a0161da0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 --2- 172.21.15.32:0/1648018874 >> v2:172.21.15.59:3300/0 conn(0x7f47a0158830 0x7f47a006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.704+0000 7f47a87ba640 1 -- 172.21.15.32:0/1648018874 >> 172.21.15.32:0/1648018874 conn(0x7f47a00c83d0 msgr2=0x7f47a00c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.708+0000 7f47a87ba640 1 -- 172.21.15.32:0/1648018874 shutdown_connections 2024-03-16T19:57:27.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.708+0000 7f47a87ba640 1 -- 172.21.15.32:0/1648018874 wait complete. 2024-03-16T19:57:27.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 Processor -- start 2024-03-16T19:57:27.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 -- start start 2024-03-16T19:57:27.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a0071650 0x7f47a0148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a00c8f30 0x7f47a01544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0154010 con 0x7f47a00c8f30 2024-03-16T19:57:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00c49f0 con 0x7f47a0071650 2024-03-16T19:57:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a87ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00caf00 con 0x7f47a0158830 2024-03-16T19:57:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a6d57640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a6d57640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44216/0 (socket says 172.21.15.32:44216) 2024-03-16T19:57:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.712+0000 7f47a6d57640 1 -- 172.21.15.32:0/3781837309 learned_addr learned my addr 172.21.15.32:0/3781837309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.716+0000 7f47a6d57640 1 -- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a0071650 msgr2=0x7f47a0148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.716+0000 7f47a6d57640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a0071650 0x7f47a0148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.716+0000 7f47a6d57640 1 -- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a00c8f30 msgr2=0x7f47a01544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.716+0000 7f47a6d57640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a00c8f30 0x7f47a01544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.716+0000 7f47a6d57640 1 -- 172.21.15.32:0/3781837309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009b30 con 0x7f47a0158830 2024-03-16T19:57:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.720+0000 7f47a6d57640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e20 tx=0x7f4798019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.728+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980b8880 con 0x7f47a0158830 2024-03-16T19:57:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.728+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980b89e0 con 0x7f47a0158830 2024-03-16T19:57:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.728+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a00cb1e0 con 0x7f47a0158830 2024-03-16T19:57:27.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.728+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980b8d90 con 0x7f47a0158830 2024-03-16T19:57:27.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.728+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0153ca0 con 0x7f47a0158830 2024-03-16T19:57:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.732+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0153820 con 0x7f47a0158830 2024-03-16T19:57:27.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.756+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f47980ca430 con 0x7f47a0158830 2024-03-16T19:57:27.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.760+0000 7f4796ffd640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 0x7f4784069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:27.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.760+0000 7f47a6556640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 0x7f4784069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:27.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.760+0000 7f47a6556640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 0x7f4784069350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4790009730 tx=0x7f4790006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:27.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.764+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f47980f1640 con 0x7f47a0158830 2024-03-16T19:57:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.820+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6fc0068000 con 0x7f6fc0150d70 2024-03-16T19:57:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.852+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc4c40f0460 con 0x7fc4cc0c7560 2024-03-16T19:57:27.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.860+0000 7f85c6ffd640 1 -- 172.21.15.32:0/2786360856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85d00c3c60 con 0x7f85d00c5f40 2024-03-16T19:57:27.830 INFO:teuthology.orchestra.run.smithi032.stdout:68719476758 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 msgr2=0x7f85b4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 0x7f85b4069350 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85d01480a0 tx=0x7f85c0009060 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 msgr2=0x7f85d0144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d5f0 tx=0x7f85cc00dab0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 shutdown_connections 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85b4066e90 0x7f85b4069350 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85d0158560 0x7f85d00c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85d00c6320 0x7f85d01453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 --2- 172.21.15.32:0/2786360856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d00c5f40 0x7f85d0144e80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 >> 172.21.15.32:0/2786360856 conn(0x7f85d00cc7a0 msgr2=0x7f85d01565f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 shutdown_connections 2024-03-16T19:57:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.864+0000 7f85d8477640 1 -- 172.21.15.32:0/2786360856 wait complete. 2024-03-16T19:57:27.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476757 got 68719476758 for osd.4 2024-03-16T19:57:27.857 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:27.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:27.916+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f052c068000 con 0x7f052c151e50 2024-03-16T19:57:27.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.012+0000 7f6fa7fff640 1 -- 172.21.15.32:0/2712690349 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6fb8058600 con 0x7f6fc0150d70 2024-03-16T19:57:27.983 INFO:teuthology.orchestra.run.smithi032.stdout:60129542166 2024-03-16T19:57:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 msgr2=0x7f6fac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 0x7f6fac069350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000a7b0 tx=0x7f6fb0008040 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 msgr2=0x7f6fc0158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0071610 tx=0x7f6fb800c8e0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 shutdown_connections 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fac066e90 0x7f6fac069350 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0150d70 0x7f6fc0158d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc014bf50 0x7f6fc014a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 --2- 172.21.15.32:0/2712690349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fa80089f0 0x7f6fc0153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 >> 172.21.15.32:0/2712690349 conn(0x7f6fc006f3a0 msgr2=0x7f6fc0071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 shutdown_connections 2024-03-16T19:57:27.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.016+0000 7f6fc6062640 1 -- 172.21.15.32:0/2712690349 wait complete. 2024-03-16T19:57:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.020+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc4cc0c6080 con 0x7fc4cc0c7560 2024-03-16T19:57:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.028+0000 7fc4c27fc640 1 -- 172.21.15.32:0/380461150 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4c40b4070 con 0x7fc4cc0c7560 2024-03-16T19:57:27.999 INFO:teuthology.orchestra.run.smithi032.stdout:68719476761 2024-03-16T19:57:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.028+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9954007450 con 0x7f9964151c30 2024-03-16T19:57:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.028+0000 7f994bfff640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 0x7f9950069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.028+0000 7f99637fe640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 0x7f9950069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.028+0000 7f99637fe640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 0x7f9950069350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f99580b2000 tx=0x7f9958004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:28.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 msgr2=0x7fc4b00692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 0x7fc4b00692e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007fd0 tx=0x7fc4c80079a0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 msgr2=0x7fc4cc0c3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40b2040 tx=0x7fc4c401bf40 comp rx=0 tx=0).stop 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 shutdown_connections 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc4b0066e20 0x7fc4b00692e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4cc0c7560 0x7fc4cc0c3f10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0c54b0 0x7fc4cc154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 --2- 172.21.15.32:0/380461150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc0b1a90 0x7fc4cc1e3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 >> 172.21.15.32:0/380461150 conn(0x7fc4cc06f3a0 msgr2=0x7fc4cc0c6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 shutdown_connections 2024-03-16T19:57:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7fc4d3ae3640 1 -- 172.21.15.32:0/380461150 wait complete. 2024-03-16T19:57:28.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.032+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9954054f30 con 0x7f9964151c30 2024-03-16T19:57:28.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542166 got 60129542166 for osd.1 2024-03-16T19:57:28.010 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:28.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.048+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f99540511d0 con 0x7f9964151c30 2024-03-16T19:57:28.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476760 got 68719476761 for osd.6 2024-03-16T19:57:28.024 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:28.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.096+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f47980ed1e0 con 0x7f47a0158830 2024-03-16T19:57:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.104+0000 7f0513fff640 1 -- 172.21.15.32:0/3024398633 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0524017070 con 0x7f052c151e50 2024-03-16T19:57:28.074 INFO:teuthology.orchestra.run.smithi032.stdout:64424509463 2024-03-16T19:57:28.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.104+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 msgr2=0x7f0518069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.104+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 0x7f0518069300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f051c009690 tx=0x7f051c006c00 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 msgr2=0x7f052c0b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f052400c780 tx=0x7f052400cc40 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 shutdown_connections 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0518066e40 0x7f0518069300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f052c151e50 0x7f052c0b23f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c0c97b0 0x7f052c0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 --2- 172.21.15.32:0/3024398633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f052c06f0a0 0x7f052c06bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 >> 172.21.15.32:0/3024398633 conn(0x7f052c070f90 msgr2=0x7f052c14cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 shutdown_connections 2024-03-16T19:57:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.108+0000 7f0531817640 1 -- 172.21.15.32:0/3024398633 wait complete. 2024-03-16T19:57:28.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509463 for osd.0 2024-03-16T19:57:28.102 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:28.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.300+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f47a0153820 con 0x7f47a0158830 2024-03-16T19:57:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f4796ffd640 1 -- 172.21.15.32:0/3781837309 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47a0153820 con 0x7f47a0158830 2024-03-16T19:57:28.278 INFO:teuthology.orchestra.run.smithi032.stdout:64424509463 2024-03-16T19:57:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 msgr2=0x7f4784069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 0x7f4784069350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4790009730 tx=0x7f4790006ca0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 msgr2=0x7f47a01549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e20 tx=0x7f4798019740 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 shutdown_connections 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4784066e90 0x7f4784069350 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f47a0158830 0x7f47a01549e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a00c8f30 0x7f47a01544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 --2- 172.21.15.32:0/3781837309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a0071650 0x7f47a0148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.308+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 >> 172.21.15.32:0/3781837309 conn(0x7f47a00c83d0 msgr2=0x7f47a00c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.312+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 shutdown_connections 2024-03-16T19:57:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.312+0000 7f47a87ba640 1 -- 172.21.15.32:0/3781837309 wait complete. 2024-03-16T19:57:28.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.328+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9944005820 con 0x7f9964151c30 2024-03-16T19:57:28.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509462 got 64424509463 for osd.2 2024-03-16T19:57:28.303 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:28.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.332+0000 7f994bfff640 1 -- 172.21.15.32:0/3176845693 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9954017070 con 0x7f9964151c30 2024-03-16T19:57:28.304 INFO:teuthology.orchestra.run.smithi032.stdout:68719476760 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 msgr2=0x7f9950069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 0x7f9950069350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f99580b2000 tx=0x7f9958004740 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 msgr2=0x7f996414ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 0x7f996414ff50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f995400c340 tx=0x7f995400c800 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 shutdown_connections 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9950066e90 0x7f9950069350 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9964151c30 0x7f996414ff50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99640c2d20 0x7f996414fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 --2- 172.21.15.32:0/3176845693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99640c2900 0x7f99640722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 >> 172.21.15.32:0/3176845693 conn(0x7f996406b190 msgr2=0x7f99640c1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 shutdown_connections 2024-03-16T19:57:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.336+0000 7f9968f3b640 1 -- 172.21.15.32:0/3176845693 wait complete. 2024-03-16T19:57:28.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476759 got 68719476760 for osd.5 2024-03-16T19:57:28.329 DEBUG:teuthology.parallel:result is None 2024-03-16T19:57:28.329 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-16T19:57:28.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:57:28.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 Processor -- start 2024-03-16T19:57:28.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 -- start start 2024-03-16T19:57:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27940c2d70 0x7f279415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279406bae0 con 0x7f2794072a10 2024-03-16T19:57:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279415a540 con 0x7f27940c2d70 2024-03-16T19:57:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.504+0000 7f279be29640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27940c7fd0 con 0x7f2794154350 2024-03-16T19:57:28.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.508+0000 7f2799bc5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.508+0000 7f279a3c6640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2794072a10 0x7f27940c1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49036/0 (socket says 172.21.15.32:49036) 2024-03-16T19:57:28.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.508+0000 7f279a3c6640 1 -- 172.21.15.32:0/3275732680 learned_addr learned my addr 172.21.15.32:0/3275732680 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:28.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.508+0000 7f27993c4640 1 --2- 172.21.15.32:0/3275732680 >> v2:172.21.15.32:3300/0 conn(0x7f27940c2d70 0x7f279415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.512+0000 7f27993c4640 1 -- 172.21.15.32:0/3275732680 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 msgr2=0x7f279406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.512+0000 7f27993c4640 1 --2- 172.21.15.32:0/3275732680 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.512+0000 7f27993c4640 1 -- 172.21.15.32:0/3275732680 >> v1:172.21.15.32:6789/0 conn(0x7f2794072a10 legacy=0x7f27940c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.512+0000 7f27993c4640 1 -- 172.21.15.32:0/3275732680 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279406bd10 con 0x7f27940c2d70 2024-03-16T19:57:28.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.516+0000 7f27993c4640 1 --2- 172.21.15.32:0/3275732680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27940c2d70 0x7f279415a9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27840097c0 tx=0x7f278401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259821be373ea4c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f2798bc3640 1 -- 172.21.15.32:0/3275732680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27840ab600 con 0x7f27940c2d70 2024-03-16T19:57:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f2798bc3640 1 -- 172.21.15.32:0/3275732680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27840a7050 con 0x7f27940c2d70 2024-03-16T19:57:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f2798bc3640 1 -- 172.21.15.32:0/3275732680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27840ab980 con 0x7f27940c2d70 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 -- 172.21.15.32:0/3275732680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27940c2d70 msgr2=0x7f279415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 --2- 172.21.15.32:0/3275732680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27940c2d70 0x7f279415a9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27840097c0 tx=0x7f278401c410 comp rx=0 tx=0).stop 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 -- 172.21.15.32:0/3275732680 shutdown_connections 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 --2- 172.21.15.32:0/3275732680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27940c2d70 0x7f279415a9d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 --2- 172.21.15.32:0/3275732680 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 -- 172.21.15.32:0/3275732680 >> 172.21.15.32:0/3275732680 conn(0x7f279406afc0 msgr2=0x7f279406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 -- 172.21.15.32:0/3275732680 shutdown_connections 2024-03-16T19:57:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.520+0000 7f279be29640 1 -- 172.21.15.32:0/3275732680 wait complete. 2024-03-16T19:57:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 Processor -- start 2024-03-16T19:57:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 -- start start 2024-03-16T19:57:28.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27940c2d70 0x7f279415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941f39d0 con 0x7f2794072a10 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279415c150 con 0x7f27940c2d70 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279be29640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27941f3430 con 0x7f2794154350 2024-03-16T19:57:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f2799bc5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f279a3c6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f2799bc5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59978/0 (socket says 172.21.15.32:59978) 2024-03-16T19:57:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f2799bc5640 1 -- 172.21.15.32:0/1781087959 learned_addr learned my addr 172.21.15.32:0/1781087959 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.524+0000 7f27993c4640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27940c2d70 0x7f279415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.532+0000 7f2799bc5640 1 -- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 msgr2=0x7f279415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.532+0000 7f2799bc5640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.532+0000 7f2799bc5640 1 -- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27940c2d70 msgr2=0x7f279415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.532+0000 7f2799bc5640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27940c2d70 0x7f279415c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.532+0000 7f2799bc5640 1 -- 172.21.15.32:0/1781087959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784009420 con 0x7f2794072a10 2024-03-16T19:57:28.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.536+0000 7f2799bc5640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f279000e870 tx=0x7f279000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790009510 con 0x7f2794072a10 2024-03-16T19:57:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279000eec0 con 0x7f2794072a10 2024-03-16T19:57:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f279be29640 1 -- 172.21.15.32:0/1781087959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27941f36f0 con 0x7f2794072a10 2024-03-16T19:57:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279000cb40 con 0x7f2794072a10 2024-03-16T19:57:28.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f279be29640 1 -- 172.21.15.32:0/1781087959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279415b6a0 con 0x7f2794072a10 2024-03-16T19:57:28.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.540+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27700052a0 con 0x7f2794072a10 2024-03-16T19:57:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.568+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f279000cca0 con 0x7f2794072a10 2024-03-16T19:57:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.568+0000 7f278a7fc640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 0x7f2778069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.568+0000 7f27993c4640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 0x7f2778069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.568+0000 7f27993c4640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 0x7f2778069300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27840ad000 tx=0x7f2784008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.852+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2790055f30 con 0x7f2794072a10 2024-03-16T19:57:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:28.852+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f27900521d0 con 0x7f2794072a10 2024-03-16T19:57:29.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.048+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2770000bd0 con 0x7f2778066e40 2024-03-16T19:57:29.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.064+0000 7f278a7fc640 1 -- 172.21.15.32:0/1781087959 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332708 (secure 0 0 0) 0x7f2770000bd0 con 0x7f2778066e40 2024-03-16T19:57:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:29.042 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T19:57:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.068+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 msgr2=0x7f2778069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.068+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 0x7f2778069300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27840ad000 tx=0x7f2784008040 comp rx=0 tx=0).stop 2024-03-16T19:57:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.068+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 msgr2=0x7f279414dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.068+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f279000e870 tx=0x7f279000ec70 comp rx=0 tx=0).stop 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 shutdown_connections 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2778066e40 0x7f2778069300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2794154350 0x7f279415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27940c2d70 0x7f279415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 --2- 172.21.15.32:0/1781087959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2794072a10 0x7f279414dfc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 >> 172.21.15.32:0/1781087959 conn(0x7f279406afc0 msgr2=0x7f27940ca450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 shutdown_connections 2024-03-16T19:57:29.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.072+0000 7f2777fff640 1 -- 172.21.15.32:0/1781087959 wait complete. 2024-03-16T19:57:29.062 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-16T19:57:28.581869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":365,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236528,"kb_used_data":3760,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738192,"statfs":{"total":773094113280,"available":772851908608,"internally_reserved":0,"allocated":3850240,"data_stored":2185202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5624,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003888"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":42,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.114321+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T19:57:24.114321+0000","last_peered":"2024-03-16T19:57:24.114321+0000","last_clean":"2024-03-16T19:57:24.114321+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T19:57:24.114321+0000","last_undegraded":"2024-03-16T19:57:24.114321+0000","last_fullsized":"2024-03-16T19:57:24.114321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784985+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T19:56:44.784985+0000","last_peered":"2024-03-16T19:56:44.784985+0000","last_clean":"2024-03-16T19:56:44.784985+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T19:56:44.784985+0000","last_undegraded":"2024-03-16T19:56:44.784985+0000","last_fullsized":"2024-03-16T19:56:44.784985+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782560+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T19:56:44.782560+0000","last_peered":"2024-03-16T19:56:44.782560+0000","last_clean":"2024-03-16T19:56:44.782560+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T19:56:44.782560+0000","last_undegraded":"2024-03-16T19:56:44.782560+0000","last_fullsized":"2024-03-16T19:56:44.782560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785841+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T19:56:44.785841+0000","last_peered":"2024-03-16T19:56:44.785841+0000","last_clean":"2024-03-16T19:56:44.785841+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T19:56:44.785841+0000","last_undegraded":"2024-03-16T19:56:44.785841+0000","last_fullsized":"2024-03-16T19:56:44.785841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880422+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T19:56:44.880422+0000","last_peered":"2024-03-16T19:56:44.880422+0000","last_clean":"2024-03-16T19:56:44.880422+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T19:56:44.880422+0000","last_undegraded":"2024-03-16T19:56:44.880422+0000","last_fullsized":"2024-03-16T19:56:44.880422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868804+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T19:56:44.868804+0000","last_peered":"2024-03-16T19:56:44.868804+0000","last_clean":"2024-03-16T19:56:44.868804+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T19:56:44.868804+0000","last_undegraded":"2024-03-16T19:56:44.868804+0000","last_fullsized":"2024-03-16T19:56:44.868804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'11","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.129984+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T19:57:24.129984+0000","last_peered":"2024-03-16T19:57:24.129984+0000","last_clean":"2024-03-16T19:57:24.129984+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T19:57:24.129984+0000","last_undegraded":"2024-03-16T19:57:24.129984+0000","last_fullsized":"2024-03-16T19:57:24.129984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883804+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T19:56:44.883804+0000","last_peered":"2024-03-16T19:56:44.883804+0000","last_clean":"2024-03-16T19:56:44.883804+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T19:56:44.883804+0000","last_undegraded":"2024-03-16T19:56:44.883804+0000","last_fullsized":"2024-03-16T19:56:44.883804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869676+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T19:56:44.869676+0000","last_peered":"2024-03-16T19:56:44.869676+0000","last_clean":"2024-03-16T19:56:44.869676+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T19:56:44.869676+0000","last_undegraded":"2024-03-16T19:56:44.869676+0000","last_fullsized":"2024-03-16T19:56:44.869676+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.127104+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T19:57:24.127104+0000","last_peered":"2024-03-16T19:57:24.127104+0000","last_clean":"2024-03-16T19:57:24.127104+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T19:57:24.127104+0000","last_undegraded":"2024-03-16T19:57:24.127104+0000","last_fullsized":"2024-03-16T19:57:24.127104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884919+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T19:56:44.884919+0000","last_peered":"2024-03-16T19:56:44.884919+0000","last_clean":"2024-03-16T19:56:44.884919+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T19:56:44.884919+0000","last_undegraded":"2024-03-16T19:56:44.884919+0000","last_fullsized":"2024-03-16T19:56:44.884919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869257+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T19:56:44.869257+0000","last_peered":"2024-03-16T19:56:44.869257+0000","last_clean":"2024-03-16T19:56:44.869257+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T19:56:44.869257+0000","last_undegraded":"2024-03-16T19:56:44.869257+0000","last_fullsized":"2024-03-16T19:56:44.869257+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880296+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T19:56:44.880296+0000","last_peered":"2024-03-16T19:56:44.880296+0000","last_clean":"2024-03-16T19:56:44.880296+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T19:56:44.880296+0000","last_undegraded":"2024-03-16T19:56:44.880296+0000","last_fullsized":"2024-03-16T19:56:44.880296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.452345+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T19:57:15.452345+0000","last_peered":"2024-03-16T19:57:15.452345+0000","last_clean":"2024-03-16T19:57:15.452345+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T19:57:15.452345+0000","last_undegraded":"2024-03-16T19:57:15.452345+0000","last_fullsized":"2024-03-16T19:57:15.452345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870662+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T19:56:44.870662+0000","last_peered":"2024-03-16T19:56:44.870662+0000","last_clean":"2024-03-16T19:56:44.870662+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T19:56:44.870662+0000","last_undegraded":"2024-03-16T19:56:44.870662+0000","last_fullsized":"2024-03-16T19:56:44.870662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783555+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T19:56:44.783555+0000","last_peered":"2024-03-16T19:56:44.783555+0000","last_clean":"2024-03-16T19:56:44.783555+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T19:56:44.783555+0000","last_undegraded":"2024-03-16T19:56:44.783555+0000","last_fullsized":"2024-03-16T19:56:44.783555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.686189+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T19:57:21.686189+0000","last_peered":"2024-03-16T19:57:21.686189+0000","last_clean":"2024-03-16T19:57:21.686189+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T19:57:21.686189+0000","last_undegraded":"2024-03-16T19:57:21.686189+0000","last_fullsized":"2024-03-16T19:57:21.686189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870742+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T19:56:44.870742+0000","last_peered":"2024-03-16T19:56:44.870742+0000","last_clean":"2024-03-16T19:56:44.870742+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T19:56:44.870742+0000","last_undegraded":"2024-03-16T19:56:44.870742+0000","last_fullsized":"2024-03-16T19:56:44.870742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783732+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T19:56:44.783732+0000","last_peered":"2024-03-16T19:56:44.783732+0000","last_clean":"2024-03-16T19:56:44.783732+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T19:56:44.783732+0000","last_undegraded":"2024-03-16T19:56:44.783732+0000","last_fullsized":"2024-03-16T19:56:44.783732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.225880+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T19:57:24.225880+0000","last_peered":"2024-03-16T19:57:24.225880+0000","last_clean":"2024-03-16T19:57:24.225880+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T19:57:24.225880+0000","last_undegraded":"2024-03-16T19:57:24.225880+0000","last_fullsized":"2024-03-16T19:57:24.225880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856952+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T19:56:44.856952+0000","last_peered":"2024-03-16T19:56:44.856952+0000","last_clean":"2024-03-16T19:56:44.856952+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T19:56:44.856952+0000","last_undegraded":"2024-03-16T19:56:44.856952+0000","last_fullsized":"2024-03-16T19:56:44.856952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868777+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T19:56:44.868777+0000","last_peered":"2024-03-16T19:56:44.868777+0000","last_clean":"2024-03-16T19:56:44.868777+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T19:56:44.868777+0000","last_undegraded":"2024-03-16T19:56:44.868777+0000","last_fullsized":"2024-03-16T19:56:44.868777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.454301+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T19:57:15.454301+0000","last_peered":"2024-03-16T19:57:15.454301+0000","last_clean":"2024-03-16T19:57:15.454301+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T19:57:15.454301+0000","last_undegraded":"2024-03-16T19:57:15.454301+0000","last_fullsized":"2024-03-16T19:57:15.454301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870617+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T19:56:44.870617+0000","last_peered":"2024-03-16T19:56:44.870617+0000","last_clean":"2024-03-16T19:56:44.870617+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T19:56:44.870617+0000","last_undegraded":"2024-03-16T19:56:44.870617+0000","last_fullsized":"2024-03-16T19:56:44.870617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881303+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T19:56:44.881303+0000","last_peered":"2024-03-16T19:56:44.881303+0000","last_clean":"2024-03-16T19:56:44.881303+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T19:56:44.881303+0000","last_undegraded":"2024-03-16T19:56:44.881303+0000","last_fullsized":"2024-03-16T19:56:44.881303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.249998+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T19:57:24.249998+0000","last_peered":"2024-03-16T19:57:24.249998+0000","last_clean":"2024-03-16T19:57:24.249998+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T19:57:24.249998+0000","last_undegraded":"2024-03-16T19:57:24.249998+0000","last_fullsized":"2024-03-16T19:57:24.249998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868298+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T19:56:44.868298+0000","last_peered":"2024-03-16T19:56:44.868298+0000","last_clean":"2024-03-16T19:56:44.868298+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T19:56:44.868298+0000","last_undegraded":"2024-03-16T19:56:44.868298+0000","last_fullsized":"2024-03-16T19:56:44.868298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785469+0000","last_change":"2024-03-16T19:56:36.217377+0000","last_active":"2024-03-16T19:56:44.785469+0000","last_peered":"2024-03-16T19:56:44.785469+0000","last_clean":"2024-03-16T19:56:44.785469+0000","last_became_active":"2024-03-16T19:56:36.216579+0000","last_became_peered":"2024-03-16T19:56:36.216579+0000","last_unstale":"2024-03-16T19:56:44.785469+0000","last_undegraded":"2024-03-16T19:56:44.785469+0000","last_fullsized":"2024-03-16T19:56:44.785469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.688515+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T19:57:21.688515+0000","last_peered":"2024-03-16T19:57:21.688515+0000","last_clean":"2024-03-16T19:57:21.688515+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T19:57:21.688515+0000","last_undegraded":"2024-03-16T19:57:21.688515+0000","last_fullsized":"2024-03-16T19:57:21.688515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782416+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T19:56:44.782416+0000","last_peered":"2024-03-16T19:56:44.782416+0000","last_clean":"2024-03-16T19:56:44.782416+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T19:56:44.782416+0000","last_undegraded":"2024-03-16T19:56:44.782416+0000","last_fullsized":"2024-03-16T19:56:44.782416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856282+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T19:56:44.856282+0000","last_peered":"2024-03-16T19:56:44.856282+0000","last_clean":"2024-03-16T19:56:44.856282+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T19:56:44.856282+0000","last_undegraded":"2024-03-16T19:56:44.856282+0000","last_fullsized":"2024-03-16T19:56:44.856282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.685848+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T19:57:21.685848+0000","last_peered":"2024-03-16T19:57:21.685848+0000","last_clean":"2024-03-16T19:57:21.685848+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T19:57:21.685848+0000","last_undegraded":"2024-03-16T19:57:21.685848+0000","last_fullsized":"2024-03-16T19:57:21.685848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880177+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T19:56:44.880177+0000","last_peered":"2024-03-16T19:56:44.880177+0000","last_clean":"2024-03-16T19:56:44.880177+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T19:56:44.880177+0000","last_undegraded":"2024-03-16T19:56:44.880177+0000","last_fullsized":"2024-03-16T19:56:44.880177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885381+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T19:56:44.885381+0000","last_peered":"2024-03-16T19:56:44.885381+0000","last_clean":"2024-03-16T19:56:44.885381+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T19:56:44.885381+0000","last_undegraded":"2024-03-16T19:56:44.885381+0000","last_fullsized":"2024-03-16T19:56:44.885381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856602+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T19:56:44.856602+0000","last_peered":"2024-03-16T19:56:44.856602+0000","last_clean":"2024-03-16T19:56:44.856602+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T19:56:44.856602+0000","last_undegraded":"2024-03-16T19:56:44.856602+0000","last_fullsized":"2024-03-16T19:56:44.856602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855979+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T19:56:44.855979+0000","last_peered":"2024-03-16T19:56:44.855979+0000","last_clean":"2024-03-16T19:56:44.855979+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T19:56:44.855979+0000","last_undegraded":"2024-03-16T19:56:44.855979+0000","last_fullsized":"2024-03-16T19:56:44.855979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869818+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T19:56:44.869818+0000","last_peered":"2024-03-16T19:56:44.869818+0000","last_clean":"2024-03-16T19:56:44.869818+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T19:56:44.869818+0000","last_undegraded":"2024-03-16T19:56:44.869818+0000","last_fullsized":"2024-03-16T19:56:44.869818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.252275+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T19:57:24.252275+0000","last_peered":"2024-03-16T19:57:24.252275+0000","last_clean":"2024-03-16T19:57:24.252275+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T19:57:24.252275+0000","last_undegraded":"2024-03-16T19:57:24.252275+0000","last_fullsized":"2024-03-16T19:57:24.252275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857231+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T19:56:44.857231+0000","last_peered":"2024-03-16T19:56:44.857231+0000","last_clean":"2024-03-16T19:56:44.857231+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T19:56:44.857231+0000","last_undegraded":"2024-03-16T19:56:44.857231+0000","last_fullsized":"2024-03-16T19:56:44.857231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855744+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T19:56:44.855744+0000","last_peered":"2024-03-16T19:56:44.855744+0000","last_clean":"2024-03-16T19:56:44.855744+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T19:56:44.855744+0000","last_undegraded":"2024-03-16T19:56:44.855744+0000","last_fullsized":"2024-03-16T19:56:44.855744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.480421+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T19:57:18.480421+0000","last_peered":"2024-03-16T19:57:18.480421+0000","last_clean":"2024-03-16T19:57:18.480421+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T19:57:18.480421+0000","last_undegraded":"2024-03-16T19:57:18.480421+0000","last_fullsized":"2024-03-16T19:57:18.480421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857201+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T19:56:44.857201+0000","last_peered":"2024-03-16T19:56:44.857201+0000","last_clean":"2024-03-16T19:56:44.857201+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T19:56:44.857201+0000","last_undegraded":"2024-03-16T19:56:44.857201+0000","last_fullsized":"2024-03-16T19:56:44.857201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880083+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T19:56:44.880083+0000","last_peered":"2024-03-16T19:56:44.880083+0000","last_clean":"2024-03-16T19:56:44.880083+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T19:56:44.880083+0000","last_undegraded":"2024-03-16T19:56:44.880083+0000","last_fullsized":"2024-03-16T19:56:44.880083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.371288+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T19:57:18.371288+0000","last_peered":"2024-03-16T19:57:18.371288+0000","last_clean":"2024-03-16T19:57:18.371288+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T19:57:18.371288+0000","last_undegraded":"2024-03-16T19:57:18.371288+0000","last_fullsized":"2024-03-16T19:57:18.371288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881219+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T19:56:44.881219+0000","last_peered":"2024-03-16T19:56:44.881219+0000","last_clean":"2024-03-16T19:56:44.881219+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T19:56:44.881219+0000","last_undegraded":"2024-03-16T19:56:44.881219+0000","last_fullsized":"2024-03-16T19:56:44.881219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783051+0000","last_change":"2024-03-16T19:56:36.305013+0000","last_active":"2024-03-16T19:56:44.783051+0000","last_peered":"2024-03-16T19:56:44.783051+0000","last_clean":"2024-03-16T19:56:44.783051+0000","last_became_active":"2024-03-16T19:56:36.303533+0000","last_became_peered":"2024-03-16T19:56:36.303533+0000","last_unstale":"2024-03-16T19:56:44.783051+0000","last_undegraded":"2024-03-16T19:56:44.783051+0000","last_fullsized":"2024-03-16T19:56:44.783051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.643803+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T19:57:21.643803+0000","last_peered":"2024-03-16T19:57:21.643803+0000","last_clean":"2024-03-16T19:57:21.643803+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T19:57:21.643803+0000","last_undegraded":"2024-03-16T19:57:21.643803+0000","last_fullsized":"2024-03-16T19:57:21.643803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879868+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T19:56:44.879868+0000","last_peered":"2024-03-16T19:56:44.879868+0000","last_clean":"2024-03-16T19:56:44.879868+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T19:56:44.879868+0000","last_undegraded":"2024-03-16T19:56:44.879868+0000","last_fullsized":"2024-03-16T19:56:44.879868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882315+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T19:56:44.882315+0000","last_peered":"2024-03-16T19:56:44.882315+0000","last_clean":"2024-03-16T19:56:44.882315+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T19:56:44.882315+0000","last_undegraded":"2024-03-16T19:56:44.882315+0000","last_fullsized":"2024-03-16T19:56:44.882315+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884701+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T19:56:44.884701+0000","last_peered":"2024-03-16T19:56:44.884701+0000","last_clean":"2024-03-16T19:56:44.884701+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T19:56:44.884701+0000","last_undegraded":"2024-03-16T19:56:44.884701+0000","last_fullsized":"2024-03-16T19:56:44.884701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.599617+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T19:57:21.599617+0000","last_peered":"2024-03-16T19:57:21.599617+0000","last_clean":"2024-03-16T19:57:21.599617+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T19:57:21.599617+0000","last_undegraded":"2024-03-16T19:57:21.599617+0000","last_fullsized":"2024-03-16T19:57:21.599617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880005+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T19:56:44.880005+0000","last_peered":"2024-03-16T19:56:44.880005+0000","last_clean":"2024-03-16T19:56:44.880005+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T19:56:44.880005+0000","last_undegraded":"2024-03-16T19:56:44.880005+0000","last_fullsized":"2024-03-16T19:56:44.880005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884174+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T19:56:44.884174+0000","last_peered":"2024-03-16T19:56:44.884174+0000","last_clean":"2024-03-16T19:56:44.884174+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T19:56:44.884174+0000","last_undegraded":"2024-03-16T19:56:44.884174+0000","last_fullsized":"2024-03-16T19:56:44.884174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.686783+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T19:57:21.686783+0000","last_peered":"2024-03-16T19:57:21.686783+0000","last_clean":"2024-03-16T19:57:21.686783+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T19:57:21.686783+0000","last_undegraded":"2024-03-16T19:57:21.686783+0000","last_fullsized":"2024-03-16T19:57:21.686783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880850+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T19:56:44.880850+0000","last_peered":"2024-03-16T19:56:44.880850+0000","last_clean":"2024-03-16T19:56:44.880850+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T19:56:44.880850+0000","last_undegraded":"2024-03-16T19:56:44.880850+0000","last_fullsized":"2024-03-16T19:56:44.880850+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887369+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T19:56:44.887369+0000","last_peered":"2024-03-16T19:56:44.887369+0000","last_clean":"2024-03-16T19:56:44.887369+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T19:56:44.887369+0000","last_undegraded":"2024-03-16T19:56:44.887369+0000","last_fullsized":"2024-03-16T19:56:44.887369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.185596+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T19:57:24.185596+0000","last_peered":"2024-03-16T19:57:24.185596+0000","last_clean":"2024-03-16T19:57:24.185596+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T19:57:24.185596+0000","last_undegraded":"2024-03-16T19:57:24.185596+0000","last_fullsized":"2024-03-16T19:57:24.185596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.878982+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T19:56:44.878982+0000","last_peered":"2024-03-16T19:56:44.878982+0000","last_clean":"2024-03-16T19:56:44.878982+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T19:56:44.878982+0000","last_undegraded":"2024-03-16T19:56:44.878982+0000","last_fullsized":"2024-03-16T19:56:44.878982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.886605+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T19:56:44.886605+0000","last_peered":"2024-03-16T19:56:44.886605+0000","last_clean":"2024-03-16T19:56:44.886605+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T19:56:44.886605+0000","last_undegraded":"2024-03-16T19:56:44.886605+0000","last_fullsized":"2024-03-16T19:56:44.886605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869965+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T19:56:44.869965+0000","last_peered":"2024-03-16T19:56:44.869965+0000","last_clean":"2024-03-16T19:56:44.869965+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T19:56:44.869965+0000","last_undegraded":"2024-03-16T19:56:44.869965+0000","last_fullsized":"2024-03-16T19:56:44.869965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868365+0000","last_change":"2024-03-16T19:56:43.466274+0000","last_active":"2024-03-16T19:56:44.868365+0000","last_peered":"2024-03-16T19:56:44.868365+0000","last_clean":"2024-03-16T19:56:44.868365+0000","last_became_active":"2024-03-16T19:56:43.465280+0000","last_became_peered":"2024-03-16T19:56:43.465280+0000","last_unstale":"2024-03-16T19:56:44.868365+0000","last_undegraded":"2024-03-16T19:56:44.868365+0000","last_fullsized":"2024-03-16T19:56:44.868365+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:53.230821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881133+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T19:56:44.881133+0000","last_peered":"2024-03-16T19:56:44.881133+0000","last_clean":"2024-03-16T19:56:44.881133+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T19:56:44.881133+0000","last_undegraded":"2024-03-16T19:56:44.881133+0000","last_fullsized":"2024-03-16T19:56:44.881133+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":45,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.600026+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T19:57:21.600026+0000","last_peered":"2024-03-16T19:57:21.600026+0000","last_clean":"2024-03-16T19:57:21.600026+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T19:57:21.600026+0000","last_undegraded":"2024-03-16T19:57:21.600026+0000","last_fullsized":"2024-03-16T19:57:21.600026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881060+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T19:56:44.881060+0000","last_peered":"2024-03-16T19:56:44.881060+0000","last_clean":"2024-03-16T19:56:44.881060+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T19:56:44.881060+0000","last_undegraded":"2024-03-16T19:56:44.881060+0000","last_fullsized":"2024-03-16T19:56:44.881060+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879400+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T19:56:44.879400+0000","last_peered":"2024-03-16T19:56:44.879400+0000","last_clean":"2024-03-16T19:56:44.879400+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T19:56:44.879400+0000","last_undegraded":"2024-03-16T19:56:44.879400+0000","last_fullsized":"2024-03-16T19:56:44.879400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.589222+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T19:57:21.589222+0000","last_peered":"2024-03-16T19:57:21.589222+0000","last_clean":"2024-03-16T19:57:21.589222+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T19:57:21.589222+0000","last_undegraded":"2024-03-16T19:57:21.589222+0000","last_fullsized":"2024-03-16T19:57:21.589222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784460+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T19:56:44.784460+0000","last_peered":"2024-03-16T19:56:44.784460+0000","last_clean":"2024-03-16T19:56:44.784460+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T19:56:44.784460+0000","last_undegraded":"2024-03-16T19:56:44.784460+0000","last_fullsized":"2024-03-16T19:56:44.784460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882782+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T19:56:44.882782+0000","last_peered":"2024-03-16T19:56:44.882782+0000","last_clean":"2024-03-16T19:56:44.882782+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T19:56:44.882782+0000","last_undegraded":"2024-03-16T19:56:44.882782+0000","last_fullsized":"2024-03-16T19:56:44.882782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.668975+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T19:57:21.668975+0000","last_peered":"2024-03-16T19:57:21.668975+0000","last_clean":"2024-03-16T19:57:21.668975+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T19:57:21.668975+0000","last_undegraded":"2024-03-16T19:57:21.668975+0000","last_fullsized":"2024-03-16T19:57:21.668975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782625+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T19:56:44.782625+0000","last_peered":"2024-03-16T19:56:44.782625+0000","last_clean":"2024-03-16T19:56:44.782625+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T19:56:44.782625+0000","last_undegraded":"2024-03-16T19:56:44.782625+0000","last_fullsized":"2024-03-16T19:56:44.782625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881525+0000","last_change":"2024-03-16T19:56:36.323272+0000","last_active":"2024-03-16T19:56:44.881525+0000","last_peered":"2024-03-16T19:56:44.881525+0000","last_clean":"2024-03-16T19:56:44.881525+0000","last_became_active":"2024-03-16T19:56:36.322243+0000","last_became_peered":"2024-03-16T19:56:36.322243+0000","last_unstale":"2024-03-16T19:56:44.881525+0000","last_undegraded":"2024-03-16T19:56:44.881525+0000","last_fullsized":"2024-03-16T19:56:44.881525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:49:15.281453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":38,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.114357+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T19:57:24.114357+0000","last_peered":"2024-03-16T19:57:24.114357+0000","last_clean":"2024-03-16T19:57:24.114357+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T19:57:24.114357+0000","last_undegraded":"2024-03-16T19:57:24.114357+0000","last_fullsized":"2024-03-16T19:57:24.114357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885201+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T19:56:44.885201+0000","last_peered":"2024-03-16T19:56:44.885201+0000","last_clean":"2024-03-16T19:56:44.885201+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T19:56:44.885201+0000","last_undegraded":"2024-03-16T19:56:44.885201+0000","last_fullsized":"2024-03-16T19:56:44.885201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884450+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T19:56:44.884450+0000","last_peered":"2024-03-16T19:56:44.884450+0000","last_clean":"2024-03-16T19:56:44.884450+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T19:56:44.884450+0000","last_undegraded":"2024-03-16T19:56:44.884450+0000","last_fullsized":"2024-03-16T19:56:44.884450+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":43,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.247836+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T19:57:24.247836+0000","last_peered":"2024-03-16T19:57:24.247836+0000","last_clean":"2024-03-16T19:57:24.247836+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T19:57:24.247836+0000","last_undegraded":"2024-03-16T19:57:24.247836+0000","last_fullsized":"2024-03-16T19:57:24.247836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856372+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T19:56:44.856372+0000","last_peered":"2024-03-16T19:56:44.856372+0000","last_clean":"2024-03-16T19:56:44.856372+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T19:56:44.856372+0000","last_undegraded":"2024-03-16T19:56:44.856372+0000","last_fullsized":"2024-03-16T19:56:44.856372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871108+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T19:56:44.871108+0000","last_peered":"2024-03-16T19:56:44.871108+0000","last_clean":"2024-03-16T19:56:44.871108+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T19:56:44.871108+0000","last_undegraded":"2024-03-16T19:56:44.871108+0000","last_fullsized":"2024-03-16T19:56:44.871108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.594512+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T19:57:15.594512+0000","last_peered":"2024-03-16T19:57:15.594512+0000","last_clean":"2024-03-16T19:57:15.594512+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T19:57:15.594512+0000","last_undegraded":"2024-03-16T19:57:15.594512+0000","last_fullsized":"2024-03-16T19:57:15.594512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869123+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T19:56:44.869123+0000","last_peered":"2024-03-16T19:56:44.869123+0000","last_clean":"2024-03-16T19:56:44.869123+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T19:56:44.869123+0000","last_undegraded":"2024-03-16T19:56:44.869123+0000","last_fullsized":"2024-03-16T19:56:44.869123+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880901+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T19:56:44.880901+0000","last_peered":"2024-03-16T19:56:44.880901+0000","last_clean":"2024-03-16T19:56:44.880901+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T19:56:44.880901+0000","last_undegraded":"2024-03-16T19:56:44.880901+0000","last_fullsized":"2024-03-16T19:56:44.880901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.243775+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T19:57:24.243775+0000","last_peered":"2024-03-16T19:57:24.243775+0000","last_clean":"2024-03-16T19:57:24.243775+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T19:57:24.243775+0000","last_undegraded":"2024-03-16T19:57:24.243775+0000","last_fullsized":"2024-03-16T19:57:24.243775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783021+0000","last_change":"2024-03-16T19:56:43.460381+0000","last_active":"2024-03-16T19:56:44.783021+0000","last_peered":"2024-03-16T19:56:44.783021+0000","last_clean":"2024-03-16T19:56:44.783021+0000","last_became_active":"2024-03-16T19:56:43.459928+0000","last_became_peered":"2024-03-16T19:56:43.459928+0000","last_unstale":"2024-03-16T19:56:44.783021+0000","last_undegraded":"2024-03-16T19:56:44.783021+0000","last_fullsized":"2024-03-16T19:56:44.783021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:06:01.829651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883713+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T19:56:44.883713+0000","last_peered":"2024-03-16T19:56:44.883713+0000","last_clean":"2024-03-16T19:56:44.883713+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T19:56:44.883713+0000","last_undegraded":"2024-03-16T19:56:44.883713+0000","last_fullsized":"2024-03-16T19:56:44.883713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.588720+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T19:57:15.588720+0000","last_peered":"2024-03-16T19:57:15.588720+0000","last_clean":"2024-03-16T19:57:15.588720+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T19:57:15.588720+0000","last_undegraded":"2024-03-16T19:57:15.588720+0000","last_fullsized":"2024-03-16T19:57:15.588720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782932+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T19:56:44.782932+0000","last_peered":"2024-03-16T19:56:44.782932+0000","last_clean":"2024-03-16T19:56:44.782932+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T19:56:44.782932+0000","last_undegraded":"2024-03-16T19:56:44.782932+0000","last_fullsized":"2024-03-16T19:56:44.782932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880633+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T19:56:44.880633+0000","last_peered":"2024-03-16T19:56:44.880633+0000","last_clean":"2024-03-16T19:56:44.880633+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T19:56:44.880633+0000","last_undegraded":"2024-03-16T19:56:44.880633+0000","last_fullsized":"2024-03-16T19:56:44.880633+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855564+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T19:56:44.855564+0000","last_peered":"2024-03-16T19:56:44.855564+0000","last_clean":"2024-03-16T19:56:44.855564+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T19:56:44.855564+0000","last_undegraded":"2024-03-16T19:56:44.855564+0000","last_fullsized":"2024-03-16T19:56:44.855564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.645277+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T19:57:21.645277+0000","last_peered":"2024-03-16T19:57:21.645277+0000","last_clean":"2024-03-16T19:57:21.645277+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T19:57:21.645277+0000","last_undegraded":"2024-03-16T19:57:21.645277+0000","last_fullsized":"2024-03-16T19:57:21.645277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881302+0000","last_change":"2024-03-16T19:56:43.461007+0000","last_active":"2024-03-16T19:56:44.881302+0000","last_peered":"2024-03-16T19:56:44.881302+0000","last_clean":"2024-03-16T19:56:44.881302+0000","last_became_active":"2024-03-16T19:56:43.460288+0000","last_became_peered":"2024-03-16T19:56:43.460288+0000","last_unstale":"2024-03-16T19:56:44.881302+0000","last_undegraded":"2024-03-16T19:56:44.881302+0000","last_fullsized":"2024-03-16T19:56:44.881302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:58.005225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783384+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T19:56:44.783384+0000","last_peered":"2024-03-16T19:56:44.783384+0000","last_clean":"2024-03-16T19:56:44.783384+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T19:56:44.783384+0000","last_undegraded":"2024-03-16T19:56:44.783384+0000","last_fullsized":"2024-03-16T19:56:44.783384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.385576+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T19:57:18.385576+0000","last_peered":"2024-03-16T19:57:18.385576+0000","last_clean":"2024-03-16T19:57:18.385576+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T19:57:18.385576+0000","last_undegraded":"2024-03-16T19:57:18.385576+0000","last_fullsized":"2024-03-16T19:57:18.385576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.590559+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T19:57:21.590559+0000","last_peered":"2024-03-16T19:57:21.590559+0000","last_clean":"2024-03-16T19:57:21.590559+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T19:57:21.590559+0000","last_undegraded":"2024-03-16T19:57:21.590559+0000","last_fullsized":"2024-03-16T19:57:21.590559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881333+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T19:56:44.881333+0000","last_peered":"2024-03-16T19:56:44.881333+0000","last_clean":"2024-03-16T19:56:44.881333+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T19:56:44.881333+0000","last_undegraded":"2024-03-16T19:56:44.881333+0000","last_fullsized":"2024-03-16T19:56:44.881333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880174+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T19:56:44.880174+0000","last_peered":"2024-03-16T19:56:44.880174+0000","last_clean":"2024-03-16T19:56:44.880174+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T19:56:44.880174+0000","last_undegraded":"2024-03-16T19:56:44.880174+0000","last_fullsized":"2024-03-16T19:56:44.880174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.585324+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T19:57:15.585324+0000","last_peered":"2024-03-16T19:57:15.585324+0000","last_clean":"2024-03-16T19:57:15.585324+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T19:57:15.585324+0000","last_undegraded":"2024-03-16T19:57:15.585324+0000","last_fullsized":"2024-03-16T19:57:15.585324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869247+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T19:56:44.869247+0000","last_peered":"2024-03-16T19:56:44.869247+0000","last_clean":"2024-03-16T19:56:44.869247+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T19:56:44.869247+0000","last_undegraded":"2024-03-16T19:56:44.869247+0000","last_fullsized":"2024-03-16T19:56:44.869247+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785098+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T19:56:44.785098+0000","last_peered":"2024-03-16T19:56:44.785098+0000","last_clean":"2024-03-16T19:56:44.785098+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T19:56:44.785098+0000","last_undegraded":"2024-03-16T19:56:44.785098+0000","last_fullsized":"2024-03-16T19:56:44.785098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879070+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T19:56:44.879070+0000","last_peered":"2024-03-16T19:56:44.879070+0000","last_clean":"2024-03-16T19:56:44.879070+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T19:56:44.879070+0000","last_undegraded":"2024-03-16T19:56:44.879070+0000","last_fullsized":"2024-03-16T19:56:44.879070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879660+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T19:56:44.879660+0000","last_peered":"2024-03-16T19:56:44.879660+0000","last_clean":"2024-03-16T19:56:44.879660+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T19:56:44.879660+0000","last_undegraded":"2024-03-16T19:56:44.879660+0000","last_fullsized":"2024-03-16T19:56:44.879660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883871+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T19:56:44.883871+0000","last_peered":"2024-03-16T19:56:44.883871+0000","last_clean":"2024-03-16T19:56:44.883871+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T19:56:44.883871+0000","last_undegraded":"2024-03-16T19:56:44.883871+0000","last_fullsized":"2024-03-16T19:56:44.883871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881090+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T19:56:44.881090+0000","last_peered":"2024-03-16T19:56:44.881090+0000","last_clean":"2024-03-16T19:56:44.881090+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T19:56:44.881090+0000","last_undegraded":"2024-03-16T19:56:44.881090+0000","last_fullsized":"2024-03-16T19:56:44.881090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784903+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T19:56:44.784903+0000","last_peered":"2024-03-16T19:56:44.784903+0000","last_clean":"2024-03-16T19:56:44.784903+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T19:56:44.784903+0000","last_undegraded":"2024-03-16T19:56:44.784903+0000","last_fullsized":"2024-03-16T19:56:44.784903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856454+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T19:56:44.856454+0000","last_peered":"2024-03-16T19:56:44.856454+0000","last_clean":"2024-03-16T19:56:44.856454+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T19:56:44.856454+0000","last_undegraded":"2024-03-16T19:56:44.856454+0000","last_fullsized":"2024-03-16T19:56:44.856454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883722+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T19:56:44.883722+0000","last_peered":"2024-03-16T19:56:44.883722+0000","last_clean":"2024-03-16T19:56:44.883722+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T19:56:44.883722+0000","last_undegraded":"2024-03-16T19:56:44.883722+0000","last_fullsized":"2024-03-16T19:56:44.883722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784862+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T19:56:44.784862+0000","last_peered":"2024-03-16T19:56:44.784862+0000","last_clean":"2024-03-16T19:56:44.784862+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T19:56:44.784862+0000","last_undegraded":"2024-03-16T19:56:44.784862+0000","last_fullsized":"2024-03-16T19:56:44.784862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880645+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T19:56:44.880645+0000","last_peered":"2024-03-16T19:56:44.880645+0000","last_clean":"2024-03-16T19:56:44.880645+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T19:56:44.880645+0000","last_undegraded":"2024-03-16T19:56:44.880645+0000","last_fullsized":"2024-03-16T19:56:44.880645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783243+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T19:56:44.783243+0000","last_peered":"2024-03-16T19:56:44.783243+0000","last_clean":"2024-03-16T19:56:44.783243+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T19:56:44.783243+0000","last_undegraded":"2024-03-16T19:56:44.783243+0000","last_fullsized":"2024-03-16T19:56:44.783243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883489+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T19:56:44.883489+0000","last_peered":"2024-03-16T19:56:44.883489+0000","last_clean":"2024-03-16T19:56:44.883489+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T19:56:44.883489+0000","last_undegraded":"2024-03-16T19:56:44.883489+0000","last_fullsized":"2024-03-16T19:56:44.883489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880427+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T19:56:44.880427+0000","last_peered":"2024-03-16T19:56:44.880427+0000","last_clean":"2024-03-16T19:56:44.880427+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T19:56:44.880427+0000","last_undegraded":"2024-03-16T19:56:44.880427+0000","last_fullsized":"2024-03-16T19:56:44.880427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881350+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T19:56:44.881350+0000","last_peered":"2024-03-16T19:56:44.881350+0000","last_clean":"2024-03-16T19:56:44.881350+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T19:56:44.881350+0000","last_undegraded":"2024-03-16T19:56:44.881350+0000","last_fullsized":"2024-03-16T19:56:44.881350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885426+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T19:56:44.885426+0000","last_peered":"2024-03-16T19:56:44.885426+0000","last_clean":"2024-03-16T19:56:44.885426+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T19:56:44.885426+0000","last_undegraded":"2024-03-16T19:56:44.885426+0000","last_fullsized":"2024-03-16T19:56:44.885426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880885+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T19:56:44.880885+0000","last_peered":"2024-03-16T19:56:44.880885+0000","last_clean":"2024-03-16T19:56:44.880885+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T19:56:44.880885+0000","last_undegraded":"2024-03-16T19:56:44.880885+0000","last_fullsized":"2024-03-16T19:56:44.880885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885871+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T19:56:44.885871+0000","last_peered":"2024-03-16T19:56:44.885871+0000","last_clean":"2024-03-16T19:56:44.885871+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T19:56:44.885871+0000","last_undegraded":"2024-03-16T19:56:44.885871+0000","last_fullsized":"2024-03-16T19:56:44.885871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880278+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T19:56:44.880278+0000","last_peered":"2024-03-16T19:56:44.880278+0000","last_clean":"2024-03-16T19:56:44.880278+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T19:56:44.880278+0000","last_undegraded":"2024-03-16T19:56:44.880278+0000","last_fullsized":"2024-03-16T19:56:44.880278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855681+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T19:56:44.855681+0000","last_peered":"2024-03-16T19:56:44.855681+0000","last_clean":"2024-03-16T19:56:44.855681+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T19:56:44.855681+0000","last_undegraded":"2024-03-16T19:56:44.855681+0000","last_fullsized":"2024-03-16T19:56:44.855681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881487+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T19:56:44.881487+0000","last_peered":"2024-03-16T19:56:44.881487+0000","last_clean":"2024-03-16T19:56:44.881487+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T19:56:44.881487+0000","last_undegraded":"2024-03-16T19:56:44.881487+0000","last_fullsized":"2024-03-16T19:56:44.881487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783246+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T19:56:44.783246+0000","last_peered":"2024-03-16T19:56:44.783246+0000","last_clean":"2024-03-16T19:56:44.783246+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T19:56:44.783246+0000","last_undegraded":"2024-03-16T19:56:44.783246+0000","last_fullsized":"2024-03-16T19:56:44.783246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871059+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T19:56:44.871059+0000","last_peered":"2024-03-16T19:56:44.871059+0000","last_clean":"2024-03-16T19:56:44.871059+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T19:56:44.871059+0000","last_undegraded":"2024-03-16T19:56:44.871059+0000","last_fullsized":"2024-03-16T19:56:44.871059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882578+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T19:56:44.882578+0000","last_peered":"2024-03-16T19:56:44.882578+0000","last_clean":"2024-03-16T19:56:44.882578+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T19:56:44.882578+0000","last_undegraded":"2024-03-16T19:56:44.882578+0000","last_fullsized":"2024-03-16T19:56:44.882578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883483+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T19:56:44.883483+0000","last_peered":"2024-03-16T19:56:44.883483+0000","last_clean":"2024-03-16T19:56:44.883483+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T19:56:44.883483+0000","last_undegraded":"2024-03-16T19:56:44.883483+0000","last_fullsized":"2024-03-16T19:56:44.883483+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887376+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T19:56:44.887376+0000","last_peered":"2024-03-16T19:56:44.887376+0000","last_clean":"2024-03-16T19:56:44.887376+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T19:56:44.887376+0000","last_undegraded":"2024-03-16T19:56:44.887376+0000","last_fullsized":"2024-03-16T19:56:44.887376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784039+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T19:56:44.784039+0000","last_peered":"2024-03-16T19:56:44.784039+0000","last_clean":"2024-03-16T19:56:44.784039+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T19:56:44.784039+0000","last_undegraded":"2024-03-16T19:56:44.784039+0000","last_fullsized":"2024-03-16T19:56:44.784039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880860+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T19:56:44.880860+0000","last_peered":"2024-03-16T19:56:44.880860+0000","last_clean":"2024-03-16T19:56:44.880860+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T19:56:44.880860+0000","last_undegraded":"2024-03-16T19:56:44.880860+0000","last_fullsized":"2024-03-16T19:56:44.880860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883261+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T19:56:44.883261+0000","last_peered":"2024-03-16T19:56:44.883261+0000","last_clean":"2024-03-16T19:56:44.883261+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T19:56:44.883261+0000","last_undegraded":"2024-03-16T19:56:44.883261+0000","last_fullsized":"2024-03-16T19:56:44.883261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882289+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T19:56:44.882289+0000","last_peered":"2024-03-16T19:56:44.882289+0000","last_clean":"2024-03-16T19:56:44.882289+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T19:56:44.882289+0000","last_undegraded":"2024-03-16T19:56:44.882289+0000","last_fullsized":"2024-03-16T19:56:44.882289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883418+0000","last_change":"2024-03-16T19:56:36.306936+0000","last_active":"2024-03-16T19:56:44.883418+0000","last_peered":"2024-03-16T19:56:44.883418+0000","last_clean":"2024-03-16T19:56:44.883418+0000","last_became_active":"2024-03-16T19:56:36.306436+0000","last_became_peered":"2024-03-16T19:56:36.306436+0000","last_unstale":"2024-03-16T19:56:44.883418+0000","last_undegraded":"2024-03-16T19:56:44.883418+0000","last_fullsized":"2024-03-16T19:56:44.883418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879751+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T19:56:44.879751+0000","last_peered":"2024-03-16T19:56:44.879751+0000","last_clean":"2024-03-16T19:56:44.879751+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T19:56:44.879751+0000","last_undegraded":"2024-03-16T19:56:44.879751+0000","last_fullsized":"2024-03-16T19:56:44.879751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856790+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T19:56:44.856790+0000","last_peered":"2024-03-16T19:56:44.856790+0000","last_clean":"2024-03-16T19:56:44.856790+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T19:56:44.856790+0000","last_undegraded":"2024-03-16T19:56:44.856790+0000","last_fullsized":"2024-03-16T19:56:44.856790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887093+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T19:56:44.887093+0000","last_peered":"2024-03-16T19:56:44.887093+0000","last_clean":"2024-03-16T19:56:44.887093+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T19:56:44.887093+0000","last_undegraded":"2024-03-16T19:56:44.887093+0000","last_fullsized":"2024-03-16T19:56:44.887093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476759,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":921600,"data_stored":714944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476761,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":127870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476760,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":125966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476758,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":124568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509462,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":335872,"data_stored":127904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509464,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":335872,"data_stored":124700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339760,"statfs":{"total":96636764160,"available":96603914240,"internally_reserved":0,"allocated":925696,"data_stored":714480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":124770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:57:29.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:57:29.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 Processor -- start 2024-03-16T19:57:29.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 -- start start 2024-03-16T19:57:29.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feac406ba60 0x7feac40c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4070890 con 0x7feac4151df0 2024-03-16T19:57:29.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4073370 con 0x7feac40c6320 2024-03-16T19:57:29.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.240+0000 7feacab92640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac40c6770 con 0x7feac406ba60 2024-03-16T19:57:29.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.248+0000 7feac892e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feac4151df0 0x7feac4072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36414/0 (socket says 172.21.15.32:36414) 2024-03-16T19:57:29.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.248+0000 7feac892e640 1 -- 172.21.15.32:0/944042087 learned_addr learned my addr 172.21.15.32:0/944042087 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856666361 0 0) 0x7feac4070890 con 0x7feac4151df0 2024-03-16T19:57:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4003530 con 0x7feac4151df0 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903817167 0 0) 0x7feab4003530 con 0x7feac4151df0 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 >> v2:172.21.15.59:3300/0 conn(0x7feac406ba60 msgr2=0x7feac40c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 --2- 172.21.15.32:0/944042087 >> v2:172.21.15.59:3300/0 conn(0x7feac406ba60 0x7feac40c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 >> v1:172.21.15.59:6789/0 conn(0x7feac40c6320 legacy=0x7feac4158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac406fff0 con 0x7feac4151df0 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7feab8001680 con 0x7feac4151df0 2024-03-16T19:57:29.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 >> v1:172.21.15.32:6789/0 conn(0x7feac4151df0 legacy=0x7feac4072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feab40039d0 0x7feab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab40047d0 0x7feab4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feab400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab400c1c0 con 0x7feab40052b0 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab400c330 con 0x7feab40047d0 2024-03-16T19:57:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.252+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab400c4f0 con 0x7feab40039d0 2024-03-16T19:57:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.256+0000 7feac912f640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feab400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 -- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feab40039d0 msgr2=0x7feab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feab40039d0 0x7feab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 -- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab40047d0 msgr2=0x7feab4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab40047d0 0x7feab4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 -- 172.21.15.32:0/944042087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4003530 con 0x7feab40052b0 2024-03-16T19:57:29.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.260+0000 7feac912f640 1 -- 172.21.15.32:0/944042087 reap_dead start 2024-03-16T19:57:29.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.264+0000 7feac912f640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feab400b990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feabc007fd0 tx=0x7feabc01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b3e2227cba203 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:29.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc003b80 con 0x7feab40052b0 2024-03-16T19:57:29.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc003ce0 con 0x7feab40052b0 2024-03-16T19:57:29.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feac37fe640 1 -- 172.21.15.32:0/944042087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0052b0 con 0x7feab40052b0 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 msgr2=0x7feab400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feab400b990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feabc007fd0 tx=0x7feabc01c660 comp rx=0 tx=0).stop 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- 172.21.15.32:0/944042087 shutdown_connections 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 --2- 172.21.15.32:0/944042087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feab400b990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- 172.21.15.32:0/944042087 >> 172.21.15.32:0/944042087 conn(0x7feac406c0d0 msgr2=0x7feac406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- 172.21.15.32:0/944042087 shutdown_connections 2024-03-16T19:57:29.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- 172.21.15.32:0/944042087 wait complete. 2024-03-16T19:57:29.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 Processor -- start 2024-03-16T19:57:29.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.280+0000 7feacab92640 1 -- start start 2024-03-16T19:57:29.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feac41525a0 0x7feac4153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feac4153e50 0x7feac4154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40b9040 con 0x7feab40052b0 2024-03-16T19:57:29.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40c6170 con 0x7feac4153e50 2024-03-16T19:57:29.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feacab92640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac41542d0 con 0x7feac41525a0 2024-03-16T19:57:29.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feac892e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:29.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feac892e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41866/0 (socket says 172.21.15.32:41866) 2024-03-16T19:57:29.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.284+0000 7feac892e640 1 -- 172.21.15.32:0/1578039881 learned_addr learned my addr 172.21.15.32:0/1578039881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:29.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 -- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feac41525a0 msgr2=0x7feac4153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feac41525a0 0x7feac4153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 -- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feac4153e50 msgr2=0x7feac4154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feac4153e50 0x7feac4154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 -- 172.21.15.32:0/1578039881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac41f33c0 con 0x7feab40052b0 2024-03-16T19:57:29.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.288+0000 7feac892e640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feab800c7d0 tx=0x7feab800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:29.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80074e0 con 0x7feab40052b0 2024-03-16T19:57:29.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8007640 con 0x7feab40052b0 2024-03-16T19:57:29.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac41f3680 con 0x7feab40052b0 2024-03-16T19:57:29.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8004a10 con 0x7feab40052b0 2024-03-16T19:57:29.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac40c5e90 con 0x7feab40052b0 2024-03-16T19:57:29.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.296+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4068000 con 0x7feab40052b0 2024-03-16T19:57:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.320+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feab8004b70 con 0x7feab40052b0 2024-03-16T19:57:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.320+0000 7feac17fa640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 0x7feaa8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.320+0000 7feac3fff640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 0x7feaa8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:29.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.320+0000 7feac3fff640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 0x7feaa8069330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feac414a930 tx=0x7feab0006c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.608+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7feab8055950 con 0x7feab40052b0 2024-03-16T19:57:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.608+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7feab80514f0 con 0x7feab40052b0 2024-03-16T19:57:29.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.848+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feac40705b0 con 0x7feaa8066e70 2024-03-16T19:57:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.864+0000 7feac17fa640 1 -- 172.21.15.32:0/1578039881 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332708 (secure 0 0 0) 0x7feab0009280 con 0x7feaa8066e70 2024-03-16T19:57:29.837 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:29.840 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 msgr2=0x7feaa8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 0x7feaa8069330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feac414a930 tx=0x7feab0006c00 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 msgr2=0x7feac40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feab800c7d0 tx=0x7feab800cc90 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 shutdown_connections 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feaa8066e70 0x7feaa8069330 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feac4153e50 0x7feac4154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feac41525a0 0x7feac4153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 --2- 172.21.15.32:0/1578039881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab40052b0 0x7feac40b8b00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 >> 172.21.15.32:0/1578039881 conn(0x7feac406c0d0 msgr2=0x7feac414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:29.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 shutdown_connections 2024-03-16T19:57:29.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:29.868+0000 7feacab92640 1 -- 172.21.15.32:0/1578039881 wait complete. 2024-03-16T19:57:29.855 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-16T19:57:28.581869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":365,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236528,"kb_used_data":3760,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738192,"statfs":{"total":773094113280,"available":772851908608,"internally_reserved":0,"allocated":3850240,"data_stored":2185202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5624,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003888"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":42,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.114321+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T19:57:24.114321+0000","last_peered":"2024-03-16T19:57:24.114321+0000","last_clean":"2024-03-16T19:57:24.114321+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T19:57:24.114321+0000","last_undegraded":"2024-03-16T19:57:24.114321+0000","last_fullsized":"2024-03-16T19:57:24.114321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784985+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T19:56:44.784985+0000","last_peered":"2024-03-16T19:56:44.784985+0000","last_clean":"2024-03-16T19:56:44.784985+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T19:56:44.784985+0000","last_undegraded":"2024-03-16T19:56:44.784985+0000","last_fullsized":"2024-03-16T19:56:44.784985+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782560+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T19:56:44.782560+0000","last_peered":"2024-03-16T19:56:44.782560+0000","last_clean":"2024-03-16T19:56:44.782560+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T19:56:44.782560+0000","last_undegraded":"2024-03-16T19:56:44.782560+0000","last_fullsized":"2024-03-16T19:56:44.782560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785841+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T19:56:44.785841+0000","last_peered":"2024-03-16T19:56:44.785841+0000","last_clean":"2024-03-16T19:56:44.785841+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T19:56:44.785841+0000","last_undegraded":"2024-03-16T19:56:44.785841+0000","last_fullsized":"2024-03-16T19:56:44.785841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880422+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T19:56:44.880422+0000","last_peered":"2024-03-16T19:56:44.880422+0000","last_clean":"2024-03-16T19:56:44.880422+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T19:56:44.880422+0000","last_undegraded":"2024-03-16T19:56:44.880422+0000","last_fullsized":"2024-03-16T19:56:44.880422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868804+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T19:56:44.868804+0000","last_peered":"2024-03-16T19:56:44.868804+0000","last_clean":"2024-03-16T19:56:44.868804+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T19:56:44.868804+0000","last_undegraded":"2024-03-16T19:56:44.868804+0000","last_fullsized":"2024-03-16T19:56:44.868804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'11","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.129984+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T19:57:24.129984+0000","last_peered":"2024-03-16T19:57:24.129984+0000","last_clean":"2024-03-16T19:57:24.129984+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T19:57:24.129984+0000","last_undegraded":"2024-03-16T19:57:24.129984+0000","last_fullsized":"2024-03-16T19:57:24.129984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883804+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T19:56:44.883804+0000","last_peered":"2024-03-16T19:56:44.883804+0000","last_clean":"2024-03-16T19:56:44.883804+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T19:56:44.883804+0000","last_undegraded":"2024-03-16T19:56:44.883804+0000","last_fullsized":"2024-03-16T19:56:44.883804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869676+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T19:56:44.869676+0000","last_peered":"2024-03-16T19:56:44.869676+0000","last_clean":"2024-03-16T19:56:44.869676+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T19:56:44.869676+0000","last_undegraded":"2024-03-16T19:56:44.869676+0000","last_fullsized":"2024-03-16T19:56:44.869676+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.127104+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T19:57:24.127104+0000","last_peered":"2024-03-16T19:57:24.127104+0000","last_clean":"2024-03-16T19:57:24.127104+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T19:57:24.127104+0000","last_undegraded":"2024-03-16T19:57:24.127104+0000","last_fullsized":"2024-03-16T19:57:24.127104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884919+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T19:56:44.884919+0000","last_peered":"2024-03-16T19:56:44.884919+0000","last_clean":"2024-03-16T19:56:44.884919+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T19:56:44.884919+0000","last_undegraded":"2024-03-16T19:56:44.884919+0000","last_fullsized":"2024-03-16T19:56:44.884919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869257+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T19:56:44.869257+0000","last_peered":"2024-03-16T19:56:44.869257+0000","last_clean":"2024-03-16T19:56:44.869257+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T19:56:44.869257+0000","last_undegraded":"2024-03-16T19:56:44.869257+0000","last_fullsized":"2024-03-16T19:56:44.869257+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880296+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T19:56:44.880296+0000","last_peered":"2024-03-16T19:56:44.880296+0000","last_clean":"2024-03-16T19:56:44.880296+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T19:56:44.880296+0000","last_undegraded":"2024-03-16T19:56:44.880296+0000","last_fullsized":"2024-03-16T19:56:44.880296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.452345+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T19:57:15.452345+0000","last_peered":"2024-03-16T19:57:15.452345+0000","last_clean":"2024-03-16T19:57:15.452345+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T19:57:15.452345+0000","last_undegraded":"2024-03-16T19:57:15.452345+0000","last_fullsized":"2024-03-16T19:57:15.452345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870662+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T19:56:44.870662+0000","last_peered":"2024-03-16T19:56:44.870662+0000","last_clean":"2024-03-16T19:56:44.870662+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T19:56:44.870662+0000","last_undegraded":"2024-03-16T19:56:44.870662+0000","last_fullsized":"2024-03-16T19:56:44.870662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783555+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T19:56:44.783555+0000","last_peered":"2024-03-16T19:56:44.783555+0000","last_clean":"2024-03-16T19:56:44.783555+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T19:56:44.783555+0000","last_undegraded":"2024-03-16T19:56:44.783555+0000","last_fullsized":"2024-03-16T19:56:44.783555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.686189+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T19:57:21.686189+0000","last_peered":"2024-03-16T19:57:21.686189+0000","last_clean":"2024-03-16T19:57:21.686189+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T19:57:21.686189+0000","last_undegraded":"2024-03-16T19:57:21.686189+0000","last_fullsized":"2024-03-16T19:57:21.686189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870742+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T19:56:44.870742+0000","last_peered":"2024-03-16T19:56:44.870742+0000","last_clean":"2024-03-16T19:56:44.870742+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T19:56:44.870742+0000","last_undegraded":"2024-03-16T19:56:44.870742+0000","last_fullsized":"2024-03-16T19:56:44.870742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783732+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T19:56:44.783732+0000","last_peered":"2024-03-16T19:56:44.783732+0000","last_clean":"2024-03-16T19:56:44.783732+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T19:56:44.783732+0000","last_undegraded":"2024-03-16T19:56:44.783732+0000","last_fullsized":"2024-03-16T19:56:44.783732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.225880+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T19:57:24.225880+0000","last_peered":"2024-03-16T19:57:24.225880+0000","last_clean":"2024-03-16T19:57:24.225880+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T19:57:24.225880+0000","last_undegraded":"2024-03-16T19:57:24.225880+0000","last_fullsized":"2024-03-16T19:57:24.225880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856952+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T19:56:44.856952+0000","last_peered":"2024-03-16T19:56:44.856952+0000","last_clean":"2024-03-16T19:56:44.856952+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T19:56:44.856952+0000","last_undegraded":"2024-03-16T19:56:44.856952+0000","last_fullsized":"2024-03-16T19:56:44.856952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868777+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T19:56:44.868777+0000","last_peered":"2024-03-16T19:56:44.868777+0000","last_clean":"2024-03-16T19:56:44.868777+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T19:56:44.868777+0000","last_undegraded":"2024-03-16T19:56:44.868777+0000","last_fullsized":"2024-03-16T19:56:44.868777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.454301+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T19:57:15.454301+0000","last_peered":"2024-03-16T19:57:15.454301+0000","last_clean":"2024-03-16T19:57:15.454301+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T19:57:15.454301+0000","last_undegraded":"2024-03-16T19:57:15.454301+0000","last_fullsized":"2024-03-16T19:57:15.454301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.870617+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T19:56:44.870617+0000","last_peered":"2024-03-16T19:56:44.870617+0000","last_clean":"2024-03-16T19:56:44.870617+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T19:56:44.870617+0000","last_undegraded":"2024-03-16T19:56:44.870617+0000","last_fullsized":"2024-03-16T19:56:44.870617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881303+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T19:56:44.881303+0000","last_peered":"2024-03-16T19:56:44.881303+0000","last_clean":"2024-03-16T19:56:44.881303+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T19:56:44.881303+0000","last_undegraded":"2024-03-16T19:56:44.881303+0000","last_fullsized":"2024-03-16T19:56:44.881303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.249998+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T19:57:24.249998+0000","last_peered":"2024-03-16T19:57:24.249998+0000","last_clean":"2024-03-16T19:57:24.249998+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T19:57:24.249998+0000","last_undegraded":"2024-03-16T19:57:24.249998+0000","last_fullsized":"2024-03-16T19:57:24.249998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868298+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T19:56:44.868298+0000","last_peered":"2024-03-16T19:56:44.868298+0000","last_clean":"2024-03-16T19:56:44.868298+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T19:56:44.868298+0000","last_undegraded":"2024-03-16T19:56:44.868298+0000","last_fullsized":"2024-03-16T19:56:44.868298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785469+0000","last_change":"2024-03-16T19:56:36.217377+0000","last_active":"2024-03-16T19:56:44.785469+0000","last_peered":"2024-03-16T19:56:44.785469+0000","last_clean":"2024-03-16T19:56:44.785469+0000","last_became_active":"2024-03-16T19:56:36.216579+0000","last_became_peered":"2024-03-16T19:56:36.216579+0000","last_unstale":"2024-03-16T19:56:44.785469+0000","last_undegraded":"2024-03-16T19:56:44.785469+0000","last_fullsized":"2024-03-16T19:56:44.785469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.688515+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T19:57:21.688515+0000","last_peered":"2024-03-16T19:57:21.688515+0000","last_clean":"2024-03-16T19:57:21.688515+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T19:57:21.688515+0000","last_undegraded":"2024-03-16T19:57:21.688515+0000","last_fullsized":"2024-03-16T19:57:21.688515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782416+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T19:56:44.782416+0000","last_peered":"2024-03-16T19:56:44.782416+0000","last_clean":"2024-03-16T19:56:44.782416+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T19:56:44.782416+0000","last_undegraded":"2024-03-16T19:56:44.782416+0000","last_fullsized":"2024-03-16T19:56:44.782416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856282+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T19:56:44.856282+0000","last_peered":"2024-03-16T19:56:44.856282+0000","last_clean":"2024-03-16T19:56:44.856282+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T19:56:44.856282+0000","last_undegraded":"2024-03-16T19:56:44.856282+0000","last_fullsized":"2024-03-16T19:56:44.856282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.685848+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T19:57:21.685848+0000","last_peered":"2024-03-16T19:57:21.685848+0000","last_clean":"2024-03-16T19:57:21.685848+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T19:57:21.685848+0000","last_undegraded":"2024-03-16T19:57:21.685848+0000","last_fullsized":"2024-03-16T19:57:21.685848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880177+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T19:56:44.880177+0000","last_peered":"2024-03-16T19:56:44.880177+0000","last_clean":"2024-03-16T19:56:44.880177+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T19:56:44.880177+0000","last_undegraded":"2024-03-16T19:56:44.880177+0000","last_fullsized":"2024-03-16T19:56:44.880177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885381+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T19:56:44.885381+0000","last_peered":"2024-03-16T19:56:44.885381+0000","last_clean":"2024-03-16T19:56:44.885381+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T19:56:44.885381+0000","last_undegraded":"2024-03-16T19:56:44.885381+0000","last_fullsized":"2024-03-16T19:56:44.885381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856602+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T19:56:44.856602+0000","last_peered":"2024-03-16T19:56:44.856602+0000","last_clean":"2024-03-16T19:56:44.856602+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T19:56:44.856602+0000","last_undegraded":"2024-03-16T19:56:44.856602+0000","last_fullsized":"2024-03-16T19:56:44.856602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855979+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T19:56:44.855979+0000","last_peered":"2024-03-16T19:56:44.855979+0000","last_clean":"2024-03-16T19:56:44.855979+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T19:56:44.855979+0000","last_undegraded":"2024-03-16T19:56:44.855979+0000","last_fullsized":"2024-03-16T19:56:44.855979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869818+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T19:56:44.869818+0000","last_peered":"2024-03-16T19:56:44.869818+0000","last_clean":"2024-03-16T19:56:44.869818+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T19:56:44.869818+0000","last_undegraded":"2024-03-16T19:56:44.869818+0000","last_fullsized":"2024-03-16T19:56:44.869818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.252275+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T19:57:24.252275+0000","last_peered":"2024-03-16T19:57:24.252275+0000","last_clean":"2024-03-16T19:57:24.252275+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T19:57:24.252275+0000","last_undegraded":"2024-03-16T19:57:24.252275+0000","last_fullsized":"2024-03-16T19:57:24.252275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857231+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T19:56:44.857231+0000","last_peered":"2024-03-16T19:56:44.857231+0000","last_clean":"2024-03-16T19:56:44.857231+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T19:56:44.857231+0000","last_undegraded":"2024-03-16T19:56:44.857231+0000","last_fullsized":"2024-03-16T19:56:44.857231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855744+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T19:56:44.855744+0000","last_peered":"2024-03-16T19:56:44.855744+0000","last_clean":"2024-03-16T19:56:44.855744+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T19:56:44.855744+0000","last_undegraded":"2024-03-16T19:56:44.855744+0000","last_fullsized":"2024-03-16T19:56:44.855744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.480421+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T19:57:18.480421+0000","last_peered":"2024-03-16T19:57:18.480421+0000","last_clean":"2024-03-16T19:57:18.480421+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T19:57:18.480421+0000","last_undegraded":"2024-03-16T19:57:18.480421+0000","last_fullsized":"2024-03-16T19:57:18.480421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.857201+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T19:56:44.857201+0000","last_peered":"2024-03-16T19:56:44.857201+0000","last_clean":"2024-03-16T19:56:44.857201+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T19:56:44.857201+0000","last_undegraded":"2024-03-16T19:56:44.857201+0000","last_fullsized":"2024-03-16T19:56:44.857201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880083+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T19:56:44.880083+0000","last_peered":"2024-03-16T19:56:44.880083+0000","last_clean":"2024-03-16T19:56:44.880083+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T19:56:44.880083+0000","last_undegraded":"2024-03-16T19:56:44.880083+0000","last_fullsized":"2024-03-16T19:56:44.880083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.371288+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T19:57:18.371288+0000","last_peered":"2024-03-16T19:57:18.371288+0000","last_clean":"2024-03-16T19:57:18.371288+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T19:57:18.371288+0000","last_undegraded":"2024-03-16T19:57:18.371288+0000","last_fullsized":"2024-03-16T19:57:18.371288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881219+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T19:56:44.881219+0000","last_peered":"2024-03-16T19:56:44.881219+0000","last_clean":"2024-03-16T19:56:44.881219+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T19:56:44.881219+0000","last_undegraded":"2024-03-16T19:56:44.881219+0000","last_fullsized":"2024-03-16T19:56:44.881219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783051+0000","last_change":"2024-03-16T19:56:36.305013+0000","last_active":"2024-03-16T19:56:44.783051+0000","last_peered":"2024-03-16T19:56:44.783051+0000","last_clean":"2024-03-16T19:56:44.783051+0000","last_became_active":"2024-03-16T19:56:36.303533+0000","last_became_peered":"2024-03-16T19:56:36.303533+0000","last_unstale":"2024-03-16T19:56:44.783051+0000","last_undegraded":"2024-03-16T19:56:44.783051+0000","last_fullsized":"2024-03-16T19:56:44.783051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.643803+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T19:57:21.643803+0000","last_peered":"2024-03-16T19:57:21.643803+0000","last_clean":"2024-03-16T19:57:21.643803+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T19:57:21.643803+0000","last_undegraded":"2024-03-16T19:57:21.643803+0000","last_fullsized":"2024-03-16T19:57:21.643803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879868+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T19:56:44.879868+0000","last_peered":"2024-03-16T19:56:44.879868+0000","last_clean":"2024-03-16T19:56:44.879868+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T19:56:44.879868+0000","last_undegraded":"2024-03-16T19:56:44.879868+0000","last_fullsized":"2024-03-16T19:56:44.879868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882315+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T19:56:44.882315+0000","last_peered":"2024-03-16T19:56:44.882315+0000","last_clean":"2024-03-16T19:56:44.882315+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T19:56:44.882315+0000","last_undegraded":"2024-03-16T19:56:44.882315+0000","last_fullsized":"2024-03-16T19:56:44.882315+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884701+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T19:56:44.884701+0000","last_peered":"2024-03-16T19:56:44.884701+0000","last_clean":"2024-03-16T19:56:44.884701+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T19:56:44.884701+0000","last_undegraded":"2024-03-16T19:56:44.884701+0000","last_fullsized":"2024-03-16T19:56:44.884701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.599617+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T19:57:21.599617+0000","last_peered":"2024-03-16T19:57:21.599617+0000","last_clean":"2024-03-16T19:57:21.599617+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T19:57:21.599617+0000","last_undegraded":"2024-03-16T19:57:21.599617+0000","last_fullsized":"2024-03-16T19:57:21.599617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880005+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T19:56:44.880005+0000","last_peered":"2024-03-16T19:56:44.880005+0000","last_clean":"2024-03-16T19:56:44.880005+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T19:56:44.880005+0000","last_undegraded":"2024-03-16T19:56:44.880005+0000","last_fullsized":"2024-03-16T19:56:44.880005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884174+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T19:56:44.884174+0000","last_peered":"2024-03-16T19:56:44.884174+0000","last_clean":"2024-03-16T19:56:44.884174+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T19:56:44.884174+0000","last_undegraded":"2024-03-16T19:56:44.884174+0000","last_fullsized":"2024-03-16T19:56:44.884174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.686783+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T19:57:21.686783+0000","last_peered":"2024-03-16T19:57:21.686783+0000","last_clean":"2024-03-16T19:57:21.686783+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T19:57:21.686783+0000","last_undegraded":"2024-03-16T19:57:21.686783+0000","last_fullsized":"2024-03-16T19:57:21.686783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880850+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T19:56:44.880850+0000","last_peered":"2024-03-16T19:56:44.880850+0000","last_clean":"2024-03-16T19:56:44.880850+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T19:56:44.880850+0000","last_undegraded":"2024-03-16T19:56:44.880850+0000","last_fullsized":"2024-03-16T19:56:44.880850+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887369+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T19:56:44.887369+0000","last_peered":"2024-03-16T19:56:44.887369+0000","last_clean":"2024-03-16T19:56:44.887369+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T19:56:44.887369+0000","last_undegraded":"2024-03-16T19:56:44.887369+0000","last_fullsized":"2024-03-16T19:56:44.887369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.185596+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T19:57:24.185596+0000","last_peered":"2024-03-16T19:57:24.185596+0000","last_clean":"2024-03-16T19:57:24.185596+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T19:57:24.185596+0000","last_undegraded":"2024-03-16T19:57:24.185596+0000","last_fullsized":"2024-03-16T19:57:24.185596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.878982+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T19:56:44.878982+0000","last_peered":"2024-03-16T19:56:44.878982+0000","last_clean":"2024-03-16T19:56:44.878982+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T19:56:44.878982+0000","last_undegraded":"2024-03-16T19:56:44.878982+0000","last_fullsized":"2024-03-16T19:56:44.878982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.886605+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T19:56:44.886605+0000","last_peered":"2024-03-16T19:56:44.886605+0000","last_clean":"2024-03-16T19:56:44.886605+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T19:56:44.886605+0000","last_undegraded":"2024-03-16T19:56:44.886605+0000","last_fullsized":"2024-03-16T19:56:44.886605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869965+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T19:56:44.869965+0000","last_peered":"2024-03-16T19:56:44.869965+0000","last_clean":"2024-03-16T19:56:44.869965+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T19:56:44.869965+0000","last_undegraded":"2024-03-16T19:56:44.869965+0000","last_fullsized":"2024-03-16T19:56:44.869965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.868365+0000","last_change":"2024-03-16T19:56:43.466274+0000","last_active":"2024-03-16T19:56:44.868365+0000","last_peered":"2024-03-16T19:56:44.868365+0000","last_clean":"2024-03-16T19:56:44.868365+0000","last_became_active":"2024-03-16T19:56:43.465280+0000","last_became_peered":"2024-03-16T19:56:43.465280+0000","last_unstale":"2024-03-16T19:56:44.868365+0000","last_undegraded":"2024-03-16T19:56:44.868365+0000","last_fullsized":"2024-03-16T19:56:44.868365+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:53.230821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881133+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T19:56:44.881133+0000","last_peered":"2024-03-16T19:56:44.881133+0000","last_clean":"2024-03-16T19:56:44.881133+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T19:56:44.881133+0000","last_undegraded":"2024-03-16T19:56:44.881133+0000","last_fullsized":"2024-03-16T19:56:44.881133+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":45,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.600026+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T19:57:21.600026+0000","last_peered":"2024-03-16T19:57:21.600026+0000","last_clean":"2024-03-16T19:57:21.600026+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T19:57:21.600026+0000","last_undegraded":"2024-03-16T19:57:21.600026+0000","last_fullsized":"2024-03-16T19:57:21.600026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881060+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T19:56:44.881060+0000","last_peered":"2024-03-16T19:56:44.881060+0000","last_clean":"2024-03-16T19:56:44.881060+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T19:56:44.881060+0000","last_undegraded":"2024-03-16T19:56:44.881060+0000","last_fullsized":"2024-03-16T19:56:44.881060+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879400+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T19:56:44.879400+0000","last_peered":"2024-03-16T19:56:44.879400+0000","last_clean":"2024-03-16T19:56:44.879400+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T19:56:44.879400+0000","last_undegraded":"2024-03-16T19:56:44.879400+0000","last_fullsized":"2024-03-16T19:56:44.879400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.589222+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T19:57:21.589222+0000","last_peered":"2024-03-16T19:57:21.589222+0000","last_clean":"2024-03-16T19:57:21.589222+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T19:57:21.589222+0000","last_undegraded":"2024-03-16T19:57:21.589222+0000","last_fullsized":"2024-03-16T19:57:21.589222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784460+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T19:56:44.784460+0000","last_peered":"2024-03-16T19:56:44.784460+0000","last_clean":"2024-03-16T19:56:44.784460+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T19:56:44.784460+0000","last_undegraded":"2024-03-16T19:56:44.784460+0000","last_fullsized":"2024-03-16T19:56:44.784460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882782+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T19:56:44.882782+0000","last_peered":"2024-03-16T19:56:44.882782+0000","last_clean":"2024-03-16T19:56:44.882782+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T19:56:44.882782+0000","last_undegraded":"2024-03-16T19:56:44.882782+0000","last_fullsized":"2024-03-16T19:56:44.882782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.668975+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T19:57:21.668975+0000","last_peered":"2024-03-16T19:57:21.668975+0000","last_clean":"2024-03-16T19:57:21.668975+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T19:57:21.668975+0000","last_undegraded":"2024-03-16T19:57:21.668975+0000","last_fullsized":"2024-03-16T19:57:21.668975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782625+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T19:56:44.782625+0000","last_peered":"2024-03-16T19:56:44.782625+0000","last_clean":"2024-03-16T19:56:44.782625+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T19:56:44.782625+0000","last_undegraded":"2024-03-16T19:56:44.782625+0000","last_fullsized":"2024-03-16T19:56:44.782625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881525+0000","last_change":"2024-03-16T19:56:36.323272+0000","last_active":"2024-03-16T19:56:44.881525+0000","last_peered":"2024-03-16T19:56:44.881525+0000","last_clean":"2024-03-16T19:56:44.881525+0000","last_became_active":"2024-03-16T19:56:36.322243+0000","last_became_peered":"2024-03-16T19:56:36.322243+0000","last_unstale":"2024-03-16T19:56:44.881525+0000","last_undegraded":"2024-03-16T19:56:44.881525+0000","last_fullsized":"2024-03-16T19:56:44.881525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:49:15.281453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":38,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.114357+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T19:57:24.114357+0000","last_peered":"2024-03-16T19:57:24.114357+0000","last_clean":"2024-03-16T19:57:24.114357+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T19:57:24.114357+0000","last_undegraded":"2024-03-16T19:57:24.114357+0000","last_fullsized":"2024-03-16T19:57:24.114357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885201+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T19:56:44.885201+0000","last_peered":"2024-03-16T19:56:44.885201+0000","last_clean":"2024-03-16T19:56:44.885201+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T19:56:44.885201+0000","last_undegraded":"2024-03-16T19:56:44.885201+0000","last_fullsized":"2024-03-16T19:56:44.885201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.884450+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T19:56:44.884450+0000","last_peered":"2024-03-16T19:56:44.884450+0000","last_clean":"2024-03-16T19:56:44.884450+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T19:56:44.884450+0000","last_undegraded":"2024-03-16T19:56:44.884450+0000","last_fullsized":"2024-03-16T19:56:44.884450+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":43,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.247836+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T19:57:24.247836+0000","last_peered":"2024-03-16T19:57:24.247836+0000","last_clean":"2024-03-16T19:57:24.247836+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T19:57:24.247836+0000","last_undegraded":"2024-03-16T19:57:24.247836+0000","last_fullsized":"2024-03-16T19:57:24.247836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856372+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T19:56:44.856372+0000","last_peered":"2024-03-16T19:56:44.856372+0000","last_clean":"2024-03-16T19:56:44.856372+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T19:56:44.856372+0000","last_undegraded":"2024-03-16T19:56:44.856372+0000","last_fullsized":"2024-03-16T19:56:44.856372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871108+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T19:56:44.871108+0000","last_peered":"2024-03-16T19:56:44.871108+0000","last_clean":"2024-03-16T19:56:44.871108+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T19:56:44.871108+0000","last_undegraded":"2024-03-16T19:56:44.871108+0000","last_fullsized":"2024-03-16T19:56:44.871108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.594512+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T19:57:15.594512+0000","last_peered":"2024-03-16T19:57:15.594512+0000","last_clean":"2024-03-16T19:57:15.594512+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T19:57:15.594512+0000","last_undegraded":"2024-03-16T19:57:15.594512+0000","last_fullsized":"2024-03-16T19:57:15.594512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869123+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T19:56:44.869123+0000","last_peered":"2024-03-16T19:56:44.869123+0000","last_clean":"2024-03-16T19:56:44.869123+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T19:56:44.869123+0000","last_undegraded":"2024-03-16T19:56:44.869123+0000","last_fullsized":"2024-03-16T19:56:44.869123+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880901+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T19:56:44.880901+0000","last_peered":"2024-03-16T19:56:44.880901+0000","last_clean":"2024-03-16T19:56:44.880901+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T19:56:44.880901+0000","last_undegraded":"2024-03-16T19:56:44.880901+0000","last_fullsized":"2024-03-16T19:56:44.880901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:24.243775+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T19:57:24.243775+0000","last_peered":"2024-03-16T19:57:24.243775+0000","last_clean":"2024-03-16T19:57:24.243775+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T19:57:24.243775+0000","last_undegraded":"2024-03-16T19:57:24.243775+0000","last_fullsized":"2024-03-16T19:57:24.243775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783021+0000","last_change":"2024-03-16T19:56:43.460381+0000","last_active":"2024-03-16T19:56:44.783021+0000","last_peered":"2024-03-16T19:56:44.783021+0000","last_clean":"2024-03-16T19:56:44.783021+0000","last_became_active":"2024-03-16T19:56:43.459928+0000","last_became_peered":"2024-03-16T19:56:43.459928+0000","last_unstale":"2024-03-16T19:56:44.783021+0000","last_undegraded":"2024-03-16T19:56:44.783021+0000","last_fullsized":"2024-03-16T19:56:44.783021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:06:01.829651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883713+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T19:56:44.883713+0000","last_peered":"2024-03-16T19:56:44.883713+0000","last_clean":"2024-03-16T19:56:44.883713+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T19:56:44.883713+0000","last_undegraded":"2024-03-16T19:56:44.883713+0000","last_fullsized":"2024-03-16T19:56:44.883713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.588720+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T19:57:15.588720+0000","last_peered":"2024-03-16T19:57:15.588720+0000","last_clean":"2024-03-16T19:57:15.588720+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T19:57:15.588720+0000","last_undegraded":"2024-03-16T19:57:15.588720+0000","last_fullsized":"2024-03-16T19:57:15.588720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.782932+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T19:56:44.782932+0000","last_peered":"2024-03-16T19:56:44.782932+0000","last_clean":"2024-03-16T19:56:44.782932+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T19:56:44.782932+0000","last_undegraded":"2024-03-16T19:56:44.782932+0000","last_fullsized":"2024-03-16T19:56:44.782932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880633+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T19:56:44.880633+0000","last_peered":"2024-03-16T19:56:44.880633+0000","last_clean":"2024-03-16T19:56:44.880633+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T19:56:44.880633+0000","last_undegraded":"2024-03-16T19:56:44.880633+0000","last_fullsized":"2024-03-16T19:56:44.880633+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855564+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T19:56:44.855564+0000","last_peered":"2024-03-16T19:56:44.855564+0000","last_clean":"2024-03-16T19:56:44.855564+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T19:56:44.855564+0000","last_undegraded":"2024-03-16T19:56:44.855564+0000","last_fullsized":"2024-03-16T19:56:44.855564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.645277+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T19:57:21.645277+0000","last_peered":"2024-03-16T19:57:21.645277+0000","last_clean":"2024-03-16T19:57:21.645277+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T19:57:21.645277+0000","last_undegraded":"2024-03-16T19:57:21.645277+0000","last_fullsized":"2024-03-16T19:57:21.645277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881302+0000","last_change":"2024-03-16T19:56:43.461007+0000","last_active":"2024-03-16T19:56:44.881302+0000","last_peered":"2024-03-16T19:56:44.881302+0000","last_clean":"2024-03-16T19:56:44.881302+0000","last_became_active":"2024-03-16T19:56:43.460288+0000","last_became_peered":"2024-03-16T19:56:43.460288+0000","last_unstale":"2024-03-16T19:56:44.881302+0000","last_undegraded":"2024-03-16T19:56:44.881302+0000","last_fullsized":"2024-03-16T19:56:44.881302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:58.005225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783384+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T19:56:44.783384+0000","last_peered":"2024-03-16T19:56:44.783384+0000","last_clean":"2024-03-16T19:56:44.783384+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T19:56:44.783384+0000","last_undegraded":"2024-03-16T19:56:44.783384+0000","last_fullsized":"2024-03-16T19:56:44.783384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:18.385576+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T19:57:18.385576+0000","last_peered":"2024-03-16T19:57:18.385576+0000","last_clean":"2024-03-16T19:57:18.385576+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T19:57:18.385576+0000","last_undegraded":"2024-03-16T19:57:18.385576+0000","last_fullsized":"2024-03-16T19:57:18.385576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:21.590559+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T19:57:21.590559+0000","last_peered":"2024-03-16T19:57:21.590559+0000","last_clean":"2024-03-16T19:57:21.590559+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T19:57:21.590559+0000","last_undegraded":"2024-03-16T19:57:21.590559+0000","last_fullsized":"2024-03-16T19:57:21.590559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881333+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T19:56:44.881333+0000","last_peered":"2024-03-16T19:56:44.881333+0000","last_clean":"2024-03-16T19:56:44.881333+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T19:56:44.881333+0000","last_undegraded":"2024-03-16T19:56:44.881333+0000","last_fullsized":"2024-03-16T19:56:44.881333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880174+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T19:56:44.880174+0000","last_peered":"2024-03-16T19:56:44.880174+0000","last_clean":"2024-03-16T19:56:44.880174+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T19:56:44.880174+0000","last_undegraded":"2024-03-16T19:56:44.880174+0000","last_fullsized":"2024-03-16T19:56:44.880174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:57:15.585324+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T19:57:15.585324+0000","last_peered":"2024-03-16T19:57:15.585324+0000","last_clean":"2024-03-16T19:57:15.585324+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T19:57:15.585324+0000","last_undegraded":"2024-03-16T19:57:15.585324+0000","last_fullsized":"2024-03-16T19:57:15.585324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.869247+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T19:56:44.869247+0000","last_peered":"2024-03-16T19:56:44.869247+0000","last_clean":"2024-03-16T19:56:44.869247+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T19:56:44.869247+0000","last_undegraded":"2024-03-16T19:56:44.869247+0000","last_fullsized":"2024-03-16T19:56:44.869247+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.785098+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T19:56:44.785098+0000","last_peered":"2024-03-16T19:56:44.785098+0000","last_clean":"2024-03-16T19:56:44.785098+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T19:56:44.785098+0000","last_undegraded":"2024-03-16T19:56:44.785098+0000","last_fullsized":"2024-03-16T19:56:44.785098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879070+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T19:56:44.879070+0000","last_peered":"2024-03-16T19:56:44.879070+0000","last_clean":"2024-03-16T19:56:44.879070+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T19:56:44.879070+0000","last_undegraded":"2024-03-16T19:56:44.879070+0000","last_fullsized":"2024-03-16T19:56:44.879070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879660+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T19:56:44.879660+0000","last_peered":"2024-03-16T19:56:44.879660+0000","last_clean":"2024-03-16T19:56:44.879660+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T19:56:44.879660+0000","last_undegraded":"2024-03-16T19:56:44.879660+0000","last_fullsized":"2024-03-16T19:56:44.879660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883871+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T19:56:44.883871+0000","last_peered":"2024-03-16T19:56:44.883871+0000","last_clean":"2024-03-16T19:56:44.883871+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T19:56:44.883871+0000","last_undegraded":"2024-03-16T19:56:44.883871+0000","last_fullsized":"2024-03-16T19:56:44.883871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881090+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T19:56:44.881090+0000","last_peered":"2024-03-16T19:56:44.881090+0000","last_clean":"2024-03-16T19:56:44.881090+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T19:56:44.881090+0000","last_undegraded":"2024-03-16T19:56:44.881090+0000","last_fullsized":"2024-03-16T19:56:44.881090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784903+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T19:56:44.784903+0000","last_peered":"2024-03-16T19:56:44.784903+0000","last_clean":"2024-03-16T19:56:44.784903+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T19:56:44.784903+0000","last_undegraded":"2024-03-16T19:56:44.784903+0000","last_fullsized":"2024-03-16T19:56:44.784903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856454+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T19:56:44.856454+0000","last_peered":"2024-03-16T19:56:44.856454+0000","last_clean":"2024-03-16T19:56:44.856454+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T19:56:44.856454+0000","last_undegraded":"2024-03-16T19:56:44.856454+0000","last_fullsized":"2024-03-16T19:56:44.856454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883722+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T19:56:44.883722+0000","last_peered":"2024-03-16T19:56:44.883722+0000","last_clean":"2024-03-16T19:56:44.883722+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T19:56:44.883722+0000","last_undegraded":"2024-03-16T19:56:44.883722+0000","last_fullsized":"2024-03-16T19:56:44.883722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784862+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T19:56:44.784862+0000","last_peered":"2024-03-16T19:56:44.784862+0000","last_clean":"2024-03-16T19:56:44.784862+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T19:56:44.784862+0000","last_undegraded":"2024-03-16T19:56:44.784862+0000","last_fullsized":"2024-03-16T19:56:44.784862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880645+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T19:56:44.880645+0000","last_peered":"2024-03-16T19:56:44.880645+0000","last_clean":"2024-03-16T19:56:44.880645+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T19:56:44.880645+0000","last_undegraded":"2024-03-16T19:56:44.880645+0000","last_fullsized":"2024-03-16T19:56:44.880645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783243+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T19:56:44.783243+0000","last_peered":"2024-03-16T19:56:44.783243+0000","last_clean":"2024-03-16T19:56:44.783243+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T19:56:44.783243+0000","last_undegraded":"2024-03-16T19:56:44.783243+0000","last_fullsized":"2024-03-16T19:56:44.783243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883489+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T19:56:44.883489+0000","last_peered":"2024-03-16T19:56:44.883489+0000","last_clean":"2024-03-16T19:56:44.883489+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T19:56:44.883489+0000","last_undegraded":"2024-03-16T19:56:44.883489+0000","last_fullsized":"2024-03-16T19:56:44.883489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880427+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T19:56:44.880427+0000","last_peered":"2024-03-16T19:56:44.880427+0000","last_clean":"2024-03-16T19:56:44.880427+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T19:56:44.880427+0000","last_undegraded":"2024-03-16T19:56:44.880427+0000","last_fullsized":"2024-03-16T19:56:44.880427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881350+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T19:56:44.881350+0000","last_peered":"2024-03-16T19:56:44.881350+0000","last_clean":"2024-03-16T19:56:44.881350+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T19:56:44.881350+0000","last_undegraded":"2024-03-16T19:56:44.881350+0000","last_fullsized":"2024-03-16T19:56:44.881350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885426+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T19:56:44.885426+0000","last_peered":"2024-03-16T19:56:44.885426+0000","last_clean":"2024-03-16T19:56:44.885426+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T19:56:44.885426+0000","last_undegraded":"2024-03-16T19:56:44.885426+0000","last_fullsized":"2024-03-16T19:56:44.885426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880885+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T19:56:44.880885+0000","last_peered":"2024-03-16T19:56:44.880885+0000","last_clean":"2024-03-16T19:56:44.880885+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T19:56:44.880885+0000","last_undegraded":"2024-03-16T19:56:44.880885+0000","last_fullsized":"2024-03-16T19:56:44.880885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.885871+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T19:56:44.885871+0000","last_peered":"2024-03-16T19:56:44.885871+0000","last_clean":"2024-03-16T19:56:44.885871+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T19:56:44.885871+0000","last_undegraded":"2024-03-16T19:56:44.885871+0000","last_fullsized":"2024-03-16T19:56:44.885871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880278+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T19:56:44.880278+0000","last_peered":"2024-03-16T19:56:44.880278+0000","last_clean":"2024-03-16T19:56:44.880278+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T19:56:44.880278+0000","last_undegraded":"2024-03-16T19:56:44.880278+0000","last_fullsized":"2024-03-16T19:56:44.880278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.855681+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T19:56:44.855681+0000","last_peered":"2024-03-16T19:56:44.855681+0000","last_clean":"2024-03-16T19:56:44.855681+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T19:56:44.855681+0000","last_undegraded":"2024-03-16T19:56:44.855681+0000","last_fullsized":"2024-03-16T19:56:44.855681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.881487+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T19:56:44.881487+0000","last_peered":"2024-03-16T19:56:44.881487+0000","last_clean":"2024-03-16T19:56:44.881487+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T19:56:44.881487+0000","last_undegraded":"2024-03-16T19:56:44.881487+0000","last_fullsized":"2024-03-16T19:56:44.881487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.783246+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T19:56:44.783246+0000","last_peered":"2024-03-16T19:56:44.783246+0000","last_clean":"2024-03-16T19:56:44.783246+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T19:56:44.783246+0000","last_undegraded":"2024-03-16T19:56:44.783246+0000","last_fullsized":"2024-03-16T19:56:44.783246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.871059+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T19:56:44.871059+0000","last_peered":"2024-03-16T19:56:44.871059+0000","last_clean":"2024-03-16T19:56:44.871059+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T19:56:44.871059+0000","last_undegraded":"2024-03-16T19:56:44.871059+0000","last_fullsized":"2024-03-16T19:56:44.871059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882578+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T19:56:44.882578+0000","last_peered":"2024-03-16T19:56:44.882578+0000","last_clean":"2024-03-16T19:56:44.882578+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T19:56:44.882578+0000","last_undegraded":"2024-03-16T19:56:44.882578+0000","last_fullsized":"2024-03-16T19:56:44.882578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883483+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T19:56:44.883483+0000","last_peered":"2024-03-16T19:56:44.883483+0000","last_clean":"2024-03-16T19:56:44.883483+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T19:56:44.883483+0000","last_undegraded":"2024-03-16T19:56:44.883483+0000","last_fullsized":"2024-03-16T19:56:44.883483+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887376+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T19:56:44.887376+0000","last_peered":"2024-03-16T19:56:44.887376+0000","last_clean":"2024-03-16T19:56:44.887376+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T19:56:44.887376+0000","last_undegraded":"2024-03-16T19:56:44.887376+0000","last_fullsized":"2024-03-16T19:56:44.887376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.784039+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T19:56:44.784039+0000","last_peered":"2024-03-16T19:56:44.784039+0000","last_clean":"2024-03-16T19:56:44.784039+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T19:56:44.784039+0000","last_undegraded":"2024-03-16T19:56:44.784039+0000","last_fullsized":"2024-03-16T19:56:44.784039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.880860+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T19:56:44.880860+0000","last_peered":"2024-03-16T19:56:44.880860+0000","last_clean":"2024-03-16T19:56:44.880860+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T19:56:44.880860+0000","last_undegraded":"2024-03-16T19:56:44.880860+0000","last_fullsized":"2024-03-16T19:56:44.880860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883261+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T19:56:44.883261+0000","last_peered":"2024-03-16T19:56:44.883261+0000","last_clean":"2024-03-16T19:56:44.883261+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T19:56:44.883261+0000","last_undegraded":"2024-03-16T19:56:44.883261+0000","last_fullsized":"2024-03-16T19:56:44.883261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.882289+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T19:56:44.882289+0000","last_peered":"2024-03-16T19:56:44.882289+0000","last_clean":"2024-03-16T19:56:44.882289+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T19:56:44.882289+0000","last_undegraded":"2024-03-16T19:56:44.882289+0000","last_fullsized":"2024-03-16T19:56:44.882289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.883418+0000","last_change":"2024-03-16T19:56:36.306936+0000","last_active":"2024-03-16T19:56:44.883418+0000","last_peered":"2024-03-16T19:56:44.883418+0000","last_clean":"2024-03-16T19:56:44.883418+0000","last_became_active":"2024-03-16T19:56:36.306436+0000","last_became_peered":"2024-03-16T19:56:36.306436+0000","last_unstale":"2024-03-16T19:56:44.883418+0000","last_undegraded":"2024-03-16T19:56:44.883418+0000","last_fullsized":"2024-03-16T19:56:44.883418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.879751+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T19:56:44.879751+0000","last_peered":"2024-03-16T19:56:44.879751+0000","last_clean":"2024-03-16T19:56:44.879751+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T19:56:44.879751+0000","last_undegraded":"2024-03-16T19:56:44.879751+0000","last_fullsized":"2024-03-16T19:56:44.879751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.856790+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T19:56:44.856790+0000","last_peered":"2024-03-16T19:56:44.856790+0000","last_clean":"2024-03-16T19:56:44.856790+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T19:56:44.856790+0000","last_undegraded":"2024-03-16T19:56:44.856790+0000","last_fullsized":"2024-03-16T19:56:44.856790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-16T19:56:44.887093+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T19:56:44.887093+0000","last_peered":"2024-03-16T19:56:44.887093+0000","last_clean":"2024-03-16T19:56:44.887093+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T19:56:44.887093+0000","last_undegraded":"2024-03-16T19:56:44.887093+0000","last_fullsized":"2024-03-16T19:56:44.887093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476759,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":921600,"data_stored":714944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476761,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":127870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476760,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":125966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476758,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":124568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509462,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":335872,"data_stored":127904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509464,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":335872,"data_stored":124700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339760,"statfs":{"total":96636764160,"available":96603914240,"internally_reserved":0,"allocated":925696,"data_stored":714480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":124770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:57:29.861 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-16T19:57:29.861 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T19:57:29.861 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-16T19:57:29.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-16T19:57:30.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 Processor -- start 2024-03-16T19:57:30.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 -- start start 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe6800cc590 0x7fe6800cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6800cbfa0 con 0x7fe68015dbb0 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6800cceb0 con 0x7fe6800cc590 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.027+0000 7fe684dda640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68015a140 con 0x7fe68015dfc0 2024-03-16T19:57:30.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.031+0000 7fe67ed9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.031+0000 7fe67ed9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:57904/0 (socket says 172.21.15.32:57904) 2024-03-16T19:57:30.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.031+0000 7fe67ed9d640 1 -- 172.21.15.32:0/797408874 learned_addr learned my addr 172.21.15.32:0/797408874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:30.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.031+0000 7fe67e59c640 1 --2- 172.21.15.32:0/797408874 >> v2:172.21.15.59:3300/0 conn(0x7fe6800cc590 0x7fe6800cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.035+0000 7fe67ed9d640 1 -- 172.21.15.32:0/797408874 >> v2:172.21.15.59:3300/0 conn(0x7fe6800cc590 msgr2=0x7fe6800cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.035+0000 7fe67ed9d640 1 --2- 172.21.15.32:0/797408874 >> v2:172.21.15.59:3300/0 conn(0x7fe6800cc590 0x7fe6800cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.035+0000 7fe67ed9d640 1 -- 172.21.15.32:0/797408874 >> v1:172.21.15.59:6789/0 conn(0x7fe68015dbb0 legacy=0x7fe6800ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.035+0000 7fe67ed9d640 1 -- 172.21.15.32:0/797408874 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68015a3e0 con 0x7fe68015dfc0 2024-03-16T19:57:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.039+0000 7fe67ed9d640 1 --2- 172.21.15.32:0/797408874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe678009e50 tx=0x7fe67801f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb64521501071f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:30.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe67d59a640 1 -- 172.21.15.32:0/797408874 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780af400 con 0x7fe68015dfc0 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe67d59a640 1 -- 172.21.15.32:0/797408874 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6780ab050 con 0x7fe68015dfc0 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe67d59a640 1 -- 172.21.15.32:0/797408874 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780af780 con 0x7fe68015dfc0 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- 172.21.15.32:0/797408874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 msgr2=0x7fe680164740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- 172.21.15.32:0/797408874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe678009e50 tx=0x7fe67801f210 comp rx=0 tx=0).stop 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- 172.21.15.32:0/797408874 shutdown_connections 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- 172.21.15.32:0/797408874 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dfc0 0x7fe680164740 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- 172.21.15.32:0/797408874 >> v2:172.21.15.59:3300/0 conn(0x7fe6800cc590 0x7fe6800cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- 172.21.15.32:0/797408874 >> 172.21.15.32:0/797408874 conn(0x7fe680066ea0 msgr2=0x7fe6800672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- 172.21.15.32:0/797408874 shutdown_connections 2024-03-16T19:57:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- 172.21.15.32:0/797408874 wait complete. 2024-03-16T19:57:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 Processor -- start 2024-03-16T19:57:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- start start 2024-03-16T19:57:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800cc590 0x7fe6800c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dbb0 0x7fe68014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68014cb50 con 0x7fe68015dfc0 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68014c7f0 con 0x7fe6800cc590 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe684dda640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68014bac0 con 0x7fe68015dbb0 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.047+0000 7fe67ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41878/0 (socket says 172.21.15.32:41878) 2024-03-16T19:57:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 -- 172.21.15.32:0/1248706227 learned_addr learned my addr 172.21.15.32:0/1248706227 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67e59c640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800cc590 0x7fe6800c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67dd9b640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dbb0 0x7fe68014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 -- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dbb0 msgr2=0x7fe68014a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dbb0 0x7fe68014a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 -- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800cc590 msgr2=0x7fe6800c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800cc590 0x7fe6800c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.051+0000 7fe67ed9d640 1 -- 172.21.15.32:0/1248706227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe678009b30 con 0x7fe68015dfc0 2024-03-16T19:57:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.055+0000 7fe67ed9d640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe67801ce40 tx=0x7fe67801bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.059+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780afde0 con 0x7fe68015dfc0 2024-03-16T19:57:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.059+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678009510 con 0x7fe68015dfc0 2024-03-16T19:57:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.059+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68014c360 con 0x7fe68015dfc0 2024-03-16T19:57:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.059+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780ba660 con 0x7fe68015dfc0 2024-03-16T19:57:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.063+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6800c7a50 con 0x7fe68015dfc0 2024-03-16T19:57:30.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.063+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68014c4f0 con 0x7fe68015dfc0 2024-03-16T19:57:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.083+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe6780b6040 con 0x7fe68015dfc0 2024-03-16T19:57:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.087+0000 7fe66effd640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 0x7fe660069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.087+0000 7fe67e59c640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 0x7fe660069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.087+0000 7fe67e59c640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 0x7fe660069380 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6700079c0 tx=0x7fe670008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.359+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe6780f4250 con 0x7fe68015dfc0 2024-03-16T19:57:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.359+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe6780f04f0 con 0x7fe68015dfc0 2024-03-16T19:57:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.583+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe6800d03a0 con 0x7fe68015dfc0 2024-03-16T19:57:30.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.639+0000 7fe66effd640 1 -- 172.21.15.32:0/1248706227 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe6780b4040 con 0x7fe68015dfc0 2024-03-16T19:57:30.613 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:30.613 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T19:57:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 msgr2=0x7fe660069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 0x7fe660069380 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6700079c0 tx=0x7fe670008040 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 msgr2=0x7fe6800cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe67801ce40 tx=0x7fe67801bfb0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 shutdown_connections 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe660066ec0 0x7fe660069380 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68015dfc0 0x7fe6800cf250 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe68015dbb0 0x7fe68014a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 --2- 172.21.15.32:0/1248706227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800cc590 0x7fe6800c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 >> 172.21.15.32:0/1248706227 conn(0x7fe680066ea0 msgr2=0x7fe6801e3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 shutdown_connections 2024-03-16T19:57:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.643+0000 7fe684dda640 1 -- 172.21.15.32:0/1248706227 wait complete. 2024-03-16T19:57:30.637 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-16T19:57:30.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:30.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 Processor -- start 2024-03-16T19:57:30.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 -- start start 2024-03-16T19:57:30.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7facb40c36b0 0x7facb415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb40c8ec0 con 0x7facb40c8980 2024-03-16T19:57:30.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb4072f10 con 0x7facb40c36b0 2024-03-16T19:57:30.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.775+0000 7facb8dfd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40c5000 con 0x7facb4151000 2024-03-16T19:57:30.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.779+0000 7faca9d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7facb40c36b0 0x7facb415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.779+0000 7faca9d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7facb40c36b0 0x7facb415a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:39620/0 (socket says 172.21.15.32:39620) 2024-03-16T19:57:30.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.779+0000 7faca9d9b640 1 -- 172.21.15.32:0/3282313724 learned_addr learned my addr 172.21.15.32:0/3282313724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.787+0000 7faca9d9b640 1 -- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 msgr2=0x7facb40c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.787+0000 7faca9d9b640 1 --2- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.787+0000 7faca9d9b640 1 -- 172.21.15.32:0/3282313724 >> v1:172.21.15.59:6789/0 conn(0x7facb40c8980 legacy=0x7facb40c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.787+0000 7faca9d9b640 1 -- 172.21.15.32:0/3282313724 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb4072950 con 0x7facb40c36b0 2024-03-16T19:57:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.799+0000 7faca9d9b640 1 --2- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb415a240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009870 tx=0x7fac9c01c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b4042d9c2c93e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb1d9b640 1 -- 172.21.15.32:0/3282313724 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0a9620 con 0x7facb40c36b0 2024-03-16T19:57:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb1d9b640 1 -- 172.21.15.32:0/3282313724 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c0a9780 con 0x7facb40c36b0 2024-03-16T19:57:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb1d9b640 1 -- 172.21.15.32:0/3282313724 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0a9b10 con 0x7facb40c36b0 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 msgr2=0x7facb415a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb415a240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009870 tx=0x7fac9c01c560 comp rx=0 tx=0).stop 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- 172.21.15.32:0/3282313724 shutdown_connections 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb415a240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- 172.21.15.32:0/3282313724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb40c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- 172.21.15.32:0/3282313724 >> 172.21.15.32:0/3282313724 conn(0x7facb406c0d0 msgr2=0x7facb406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- 172.21.15.32:0/3282313724 shutdown_connections 2024-03-16T19:57:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- 172.21.15.32:0/3282313724 wait complete. 2024-03-16T19:57:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 Processor -- start 2024-03-16T19:57:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- start start 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb413dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb4160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb41f2840 con 0x7facb40c8980 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb41f2a30 con 0x7facb40c36b0 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.811+0000 7facb8dfd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb41f2d10 con 0x7facb4151000 2024-03-16T19:57:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.815+0000 7faca9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.815+0000 7faca9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41894/0 (socket says 172.21.15.32:41894) 2024-03-16T19:57:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.815+0000 7faca9d9b640 1 -- 172.21.15.32:0/1886594916 learned_addr learned my addr 172.21.15.32:0/1886594916 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.815+0000 7facb259c640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb413dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.819+0000 7faca9d9b640 1 -- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 msgr2=0x7facb4160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.819+0000 7faca9d9b640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb4160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.819+0000 7faca9d9b640 1 -- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 msgr2=0x7facb413dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.819+0000 7faca9d9b640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb413dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.819+0000 7faca9d9b640 1 -- 172.21.15.32:0/1886594916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c0094d0 con 0x7facb40c8980 2024-03-16T19:57:30.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.823+0000 7faca9d9b640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0ab040 tx=0x7fac9c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0bc650 con 0x7facb40c8980 2024-03-16T19:57:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c0bc7d0 con 0x7facb40c8980 2024-03-16T19:57:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0bcb80 con 0x7facb40c8980 2024-03-16T19:57:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb41f2ff0 con 0x7facb40c8980 2024-03-16T19:57:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb41f35b0 con 0x7facb40c8980 2024-03-16T19:57:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:30.827+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb4068000 con 0x7facb40c8980 2024-03-16T19:57:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.131+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fac9c0bcce0 con 0x7facb40c8980 2024-03-16T19:57:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.135+0000 7facab7fe640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 0x7fac94069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.135+0000 7facb259c640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 0x7fac94069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.135+0000 7facb259c640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 0x7fac94069330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faca40098f0 tx=0x7faca4006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.155+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fac9c0a9e30 con 0x7facb40c8980 2024-03-16T19:57:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.155+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fac9c0ec830 con 0x7facb40c8980 2024-03-16T19:57:31.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.339+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb40c62d0 con 0x7facb40c8980 2024-03-16T19:57:31.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.415+0000 7facab7fe640 1 -- 172.21.15.32:0/1886594916 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fac9c0efef0 con 0x7facb40c8980 2024-03-16T19:57:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:31.390 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 msgr2=0x7fac94069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 0x7fac94069330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faca40098f0 tx=0x7faca4006f90 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 msgr2=0x7facb4160820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0ab040 tx=0x7fac9c01ada0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 shutdown_connections 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac94066e70 0x7fac94069330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7facb4151000 0x7facb4160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facb40c8980 0x7facb4160820 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 --2- 172.21.15.32:0/1886594916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb40c36b0 0x7facb413dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 >> 172.21.15.32:0/1886594916 conn(0x7facb406c0d0 msgr2=0x7facb4142420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 shutdown_connections 2024-03-16T19:57:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.419+0000 7facb8dfd640 1 -- 172.21.15.32:0/1886594916 wait complete. 2024-03-16T19:57:31.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:56:31.637719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:56:33.839961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T19:56:40.663264+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T19:57:31.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:57:31.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 Processor -- start 2024-03-16T19:57:31.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 -- start start 2024-03-16T19:57:31.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52c415e070 0x7f52c4071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40c30e0 con 0x7f52c415d570 2024-03-16T19:57:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c4073500 con 0x7f52c4154560 2024-03-16T19:57:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52cb991640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40c2320 con 0x7f52c415e070 2024-03-16T19:57:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52c9f2e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f52c415d570 0x7f52c41648a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36418/0 (socket says 172.21.15.32:36418) 2024-03-16T19:57:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52c9f2e640 1 -- 172.21.15.32:0/4236330945 learned_addr learned my addr 172.21.15.32:0/4236330945 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:31.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.551+0000 7f52c8f2c640 1 --2- 172.21.15.32:0/4236330945 >> v2:172.21.15.59:3300/0 conn(0x7f52c415e070 0x7f52c4071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.555+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071709931 0 0) 0x7f52c40c30e0 con 0x7f52c415d570 2024-03-16T19:57:31.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.555+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0003530 con 0x7f52c415d570 2024-03-16T19:57:31.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1068269484 0 0) 0x7f52b0003530 con 0x7f52c415d570 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 >> v2:172.21.15.59:3300/0 conn(0x7f52c415e070 msgr2=0x7f52c4071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 --2- 172.21.15.32:0/4236330945 >> v2:172.21.15.59:3300/0 conn(0x7f52c415e070 0x7f52c4071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 >> v1:172.21.15.59:6789/0 conn(0x7f52c4154560 legacy=0x7f52c40c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4073220 con 0x7f52c415d570 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f52bc001a00 con 0x7f52c415d570 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 >> v1:172.21.15.32:6789/0 conn(0x7f52c415d570 legacy=0x7f52c41648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 0x7f52b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b00047d0 0x7f52b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b0005190 0x7f52b000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40c2320 con 0x7f52b00039d0 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b000c060 con 0x7f52b00047d0 2024-03-16T19:57:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.559+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b000c220 con 0x7f52b0005190 2024-03-16T19:57:31.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.563+0000 7f52c9f2e640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b0005190 0x7f52b000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.563+0000 7f52c972d640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 0x7f52b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c972d640 1 -- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b0005190 msgr2=0x7f52b000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c972d640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b0005190 0x7f52b000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c972d640 1 -- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b00047d0 msgr2=0x7f52b0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c8f2c640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b00047d0 0x7f52b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c972d640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b00047d0 0x7f52b0004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c972d640 1 -- 172.21.15.32:0/4236330945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0003530 con 0x7f52b00039d0 2024-03-16T19:57:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.567+0000 7f52c9f2e640 1 -- 172.21.15.32:0/4236330945 reap_dead start 2024-03-16T19:57:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.571+0000 7f52c972d640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 0x7f52b0003dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f52c0003db0 tx=0x7f52c001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f09b2a94f55e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00ab600 con 0x7f52b00039d0 2024-03-16T19:57:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00a7050 con 0x7f52b00039d0 2024-03-16T19:57:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52bbfff640 1 -- 172.21.15.32:0/4236330945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00ab9c0 con 0x7f52b00039d0 2024-03-16T19:57:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 -- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 msgr2=0x7f52b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 0x7f52b0003dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f52c0003db0 tx=0x7f52c001c2d0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 -- 172.21.15.32:0/4236330945 shutdown_connections 2024-03-16T19:57:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 --2- 172.21.15.32:0/4236330945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52b00039d0 0x7f52b0003dd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 -- 172.21.15.32:0/4236330945 >> 172.21.15.32:0/4236330945 conn(0x7f52c406f3a0 msgr2=0x7f52c406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 -- 172.21.15.32:0/4236330945 shutdown_connections 2024-03-16T19:57:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.575+0000 7f52cb991640 1 -- 172.21.15.32:0/4236330945 wait complete. 2024-03-16T19:57:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 Processor -- start 2024-03-16T19:57:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 -- start start 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b00039d0 0x7f52c41582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c40cb880 0x7f52c41e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40cb690 con 0x7f52c41f3370 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c41587e0 con 0x7f52c40cb880 2024-03-16T19:57:31.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52cb991640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c41f6d80 con 0x7f52b00039d0 2024-03-16T19:57:31.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52c9f2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52c9f2e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41912/0 (socket says 172.21.15.32:41912) 2024-03-16T19:57:31.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52c9f2e640 1 -- 172.21.15.32:0/2861043783 learned_addr learned my addr 172.21.15.32:0/2861043783 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:31.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52c8f2c640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c40cb880 0x7f52c41e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.579+0000 7f52c972d640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b00039d0 0x7f52c41582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.583+0000 7f52c9f2e640 1 -- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b00039d0 msgr2=0x7f52c41582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.583+0000 7f52c9f2e640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b00039d0 0x7f52c41582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.583+0000 7f52c9f2e640 1 -- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c40cb880 msgr2=0x7f52c41e8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:31.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.583+0000 7f52c9f2e640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c40cb880 0x7f52c41e8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:31.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.583+0000 7f52c9f2e640 1 -- 172.21.15.32:0/2861043783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40cb0d0 con 0x7f52c41f3370 2024-03-16T19:57:31.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.587+0000 7f52c9f2e640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f52c40671a0 tx=0x7f52bc007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.591+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00ab30 con 0x7f52c41f3370 2024-03-16T19:57:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.591+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00acb0 con 0x7f52c41f3370 2024-03-16T19:57:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.591+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc013630 con 0x7f52c41f3370 2024-03-16T19:57:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.591+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c40cb3b0 con 0x7f52c41f3370 2024-03-16T19:57:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.591+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c4066d20 con 0x7f52c41f3370 2024-03-16T19:57:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.595+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4068000 con 0x7f52c41f3370 2024-03-16T19:57:31.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.875+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f52bc013790 con 0x7f52c41f3370 2024-03-16T19:57:31.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.875+0000 7f52b9ffb640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 0x7f52a8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:31.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.875+0000 7f52c972d640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 0x7f52a8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:31.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.879+0000 7f52c972d640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 0x7f52a8069300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f52c00ad000 tx=0x7f52c0003770 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.903+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f52bc00ae20 con 0x7f52c41f3370 2024-03-16T19:57:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:31.903+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f52bc0528d0 con 0x7f52c41f3370 2024-03-16T19:57:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.063+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52c41f3020 con 0x7f52c41f3370 2024-03-16T19:57:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.075+0000 7f52b9ffb640 1 -- 172.21.15.32:0/2861043783 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52bc055f90 con 0x7f52c41f3370 2024-03-16T19:57:32.046 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:57:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 msgr2=0x7f52a8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 0x7f52a8069300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f52c00ad000 tx=0x7f52c0003770 comp rx=0 tx=0).stop 2024-03-16T19:57:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 msgr2=0x7f52c41f37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f52c40671a0 tx=0x7f52bc007bd0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 shutdown_connections 2024-03-16T19:57:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f52a8066e40 0x7f52a8069300 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c41f3370 0x7f52c41f37f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c40cb880 0x7f52c41e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 --2- 172.21.15.32:0/2861043783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f52b00039d0 0x7f52c41582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 >> 172.21.15.32:0/2861043783 conn(0x7f52c406f3a0 msgr2=0x7f52c415b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 shutdown_connections 2024-03-16T19:57:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.079+0000 7f52cb991640 1 -- 172.21.15.32:0/2861043783 wait complete. 2024-03-16T19:57:32.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:57:32.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 Processor -- start 2024-03-16T19:57:32.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 -- start start 2024-03-16T19:57:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe4d8151e50 0x7fe4d80c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4d80c97b0 0x7fe4d8157d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d815bdf0 con 0x7fe4d806f0a0 2024-03-16T19:57:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d8158d30 con 0x7fe4d80c97b0 2024-03-16T19:57:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.211+0000 7fe4de44b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d8152230 con 0x7fe4d8151e50 2024-03-16T19:57:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.275+0000 7fe4dc9e8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4d80c97b0 0x7fe4d8157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.275+0000 7fe4dc9e8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4d80c97b0 0x7fe4d8157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41928/0 (socket says 172.21.15.32:41928) 2024-03-16T19:57:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.275+0000 7fe4dc9e8640 1 -- 172.21.15.32:0/3916604102 learned_addr learned my addr 172.21.15.32:0/3916604102 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.279+0000 7fe4dc9e8640 1 -- 172.21.15.32:0/3916604102 >> v2:172.21.15.59:3300/0 conn(0x7fe4d8151e50 msgr2=0x7fe4d80c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.279+0000 7fe4dc9e8640 1 --2- 172.21.15.32:0/3916604102 >> v2:172.21.15.59:3300/0 conn(0x7fe4d8151e50 0x7fe4d80c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.279+0000 7fe4dc9e8640 1 -- 172.21.15.32:0/3916604102 >> v1:172.21.15.59:6789/0 conn(0x7fe4d806f0a0 legacy=0x7fe4d80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.279+0000 7fe4dc9e8640 1 -- 172.21.15.32:0/3916604102 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d815c280 con 0x7fe4d80c97b0 2024-03-16T19:57:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.283+0000 7fe4dc9e8640 1 --2- 172.21.15.32:0/3916604102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d80c97b0 0x7fe4d8157d30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80704f0 tx=0x7fe4d001f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b0554ac89e7518 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:32.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.283+0000 7fe4d77fe640 1 -- 172.21.15.32:0/3916604102 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00af400 con 0x7fe4d80c97b0 2024-03-16T19:57:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4d77fe640 1 -- 172.21.15.32:0/3916604102 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d00ab050 con 0x7fe4d80c97b0 2024-03-16T19:57:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4d77fe640 1 -- 172.21.15.32:0/3916604102 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00af780 con 0x7fe4d80c97b0 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 -- 172.21.15.32:0/3916604102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d80c97b0 msgr2=0x7fe4d8157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3916604102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d80c97b0 0x7fe4d8157d30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80704f0 tx=0x7fe4d001f1a0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 -- 172.21.15.32:0/3916604102 shutdown_connections 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3916604102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d80c97b0 0x7fe4d8157d30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3916604102 >> v2:172.21.15.59:3300/0 conn(0x7fe4d8151e50 0x7fe4d80c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 -- 172.21.15.32:0/3916604102 >> 172.21.15.32:0/3916604102 conn(0x7fe4d8070f90 msgr2=0x7fe4d80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 -- 172.21.15.32:0/3916604102 shutdown_connections 2024-03-16T19:57:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.287+0000 7fe4de44b640 1 -- 172.21.15.32:0/3916604102 wait complete. 2024-03-16T19:57:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 Processor -- start 2024-03-16T19:57:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 -- start start 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4d80c97b0 0x7fe4d815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4d8151e50 0x7fe4d80c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d806dad0 con 0x7fe4d806f0a0 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80c67b0 con 0x7fe4d8151e50 2024-03-16T19:57:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4de44b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80c7180 con 0x7fe4d80c97b0 2024-03-16T19:57:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4d7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4d7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41930/0 (socket says 172.21.15.32:41930) 2024-03-16T19:57:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.291+0000 7fe4d7fff640 1 -- 172.21.15.32:0/3997343513 learned_addr learned my addr 172.21.15.32:0/3997343513 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.295+0000 7fe4dc9e8640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4d8151e50 0x7fe4d80c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.299+0000 7fe4d7fff640 1 -- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4d80c97b0 msgr2=0x7fe4d815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.299+0000 7fe4d7fff640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4d80c97b0 0x7fe4d815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.299+0000 7fe4d7fff640 1 -- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4d8151e50 msgr2=0x7fe4d80c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.299+0000 7fe4d7fff640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4d8151e50 0x7fe4d80c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.299+0000 7fe4d7fff640 1 -- 172.21.15.32:0/3997343513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0009b30 con 0x7fe4d806f0a0 2024-03-16T19:57:32.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.303+0000 7fe4d7fff640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b410 tx=0x7fe4c800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011830 con 0x7fe4d806f0a0 2024-03-16T19:57:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c80119b0 con 0x7fe4d806f0a0 2024-03-16T19:57:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011d60 con 0x7fe4d806f0a0 2024-03-16T19:57:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d8157a40 con 0x7fe4d806f0a0 2024-03-16T19:57:32.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d80c6160 con 0x7fe4d806f0a0 2024-03-16T19:57:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.307+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8068000 con 0x7fe4d806f0a0 2024-03-16T19:57:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.343+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe4c801a400 con 0x7fe4d806f0a0 2024-03-16T19:57:32.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.343+0000 7fe4d57fa640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 0x7fe4b8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.347+0000 7fe4cffff640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 0x7fe4b8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:32.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.347+0000 7fe4cffff640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 0x7fe4b8069330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0007fd0 tx=0x7fe4c00079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.351+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe4c8025e30 con 0x7fe4d806f0a0 2024-03-16T19:57:32.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.687+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe4c8051f10 con 0x7fe4d806f0a0 2024-03-16T19:57:32.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.859+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4d80b3910 con 0x7fe4d806f0a0 2024-03-16T19:57:32.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.871+0000 7fe4d57fa640 1 -- 172.21.15.32:0/3997343513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4c80555d0 con 0x7fe4d806f0a0 2024-03-16T19:57:32.845 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 msgr2=0x7fe4b8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 0x7fe4b8069330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0007fd0 tx=0x7fe4c00079a0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 msgr2=0x7fe4d80c6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b410 tx=0x7fe4c800b8d0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 shutdown_connections 2024-03-16T19:57:32.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe4b8066e70 0x7fe4b8069330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4d8151e50 0x7fe4d80c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4d80c97b0 0x7fe4d815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 --2- 172.21.15.32:0/3997343513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4d806f0a0 0x7fe4d80c6c40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 >> 172.21.15.32:0/3997343513 conn(0x7fe4d8070f90 msgr2=0x7fe4d8151060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 shutdown_connections 2024-03-16T19:57:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:32.875+0000 7fe4de44b640 1 -- 172.21.15.32:0/3997343513 wait complete. 2024-03-16T19:57:32.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 Processor -- start 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 -- start start 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79000c7610 0x7f79000c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7900151410 con 0x7f790006c0d0 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79000c10a0 con 0x7f7900151030 2024-03-16T19:57:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.003+0000 7f7905ebb640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79000c7480 con 0x7f79000c7610 2024-03-16T19:57:32.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.007+0000 7f78feffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79000c7610 0x7f79000c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:32.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.007+0000 7f78fffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f790006c0d0 0x7f7900161510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36422/0 (socket says 172.21.15.32:36422) 2024-03-16T19:57:32.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.007+0000 7f78fffff640 1 -- 172.21.15.32:0/4013112396 learned_addr learned my addr 172.21.15.32:0/4013112396 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.011+0000 7f78feffd640 1 -- 172.21.15.32:0/4013112396 >> v1:172.21.15.59:6789/0 conn(0x7f7900151030 legacy=0x7f79000c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.011+0000 7f78feffd640 1 -- 172.21.15.32:0/4013112396 >> v1:172.21.15.32:6789/0 conn(0x7f790006c0d0 legacy=0x7f7900161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.011+0000 7f78feffd640 1 -- 172.21.15.32:0/4013112396 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000c1340 con 0x7f79000c7610 2024-03-16T19:57:32.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.015+0000 7f78feffd640 1 --2- 172.21.15.32:0/4013112396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79000c7610 0x7f79000c2cc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f78f40097c0 tx=0x7f78f401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9377ef5c8029e90c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f78fe7fc640 1 -- 172.21.15.32:0/4013112396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40a8990 con 0x7f79000c7610 2024-03-16T19:57:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f78fe7fc640 1 -- 172.21.15.32:0/4013112396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f40a8af0 con 0x7f79000c7610 2024-03-16T19:57:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f78fe7fc640 1 -- 172.21.15.32:0/4013112396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40a8e80 con 0x7f79000c7610 2024-03-16T19:57:32.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- 172.21.15.32:0/4013112396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79000c7610 msgr2=0x7f79000c2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4013112396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79000c7610 0x7f79000c2cc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f78f40097c0 tx=0x7f78f401ec90 comp rx=0 tx=0).stop 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- 172.21.15.32:0/4013112396 shutdown_connections 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4013112396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79000c7610 0x7f79000c2cc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- 172.21.15.32:0/4013112396 >> 172.21.15.32:0/4013112396 conn(0x7f790006f3d0 msgr2=0x7f790006f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- 172.21.15.32:0/4013112396 shutdown_connections 2024-03-16T19:57:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- 172.21.15.32:0/4013112396 wait complete. 2024-03-16T19:57:33.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 Processor -- start 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- start start 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79000c7610 0x7f79001f1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7900151030 0x7f79001f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900152700 con 0x7f7900151030 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000c3b00 con 0x7f79000c7610 2024-03-16T19:57:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.027+0000 7f7905ebb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79001f1df0 con 0x7f790006c0d0 2024-03-16T19:57:33.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.031+0000 7f78ff7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.031+0000 7f78feffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79000c7610 0x7f79001f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.031+0000 7f78ff7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:57974/0 (socket says 172.21.15.32:57974) 2024-03-16T19:57:33.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.031+0000 7f78ff7fe640 1 -- 172.21.15.32:0/4117151161 learned_addr learned my addr 172.21.15.32:0/4117151161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:33.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78fffff640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7900151030 0x7f79001f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78ff7fe640 1 -- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79000c7610 msgr2=0x7f79001f1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78ff7fe640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79000c7610 0x7f79001f1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78ff7fe640 1 -- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7900151030 msgr2=0x7f79001f17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78ff7fe640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7900151030 0x7f79001f17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.035+0000 7f78ff7fe640 1 -- 172.21.15.32:0/4117151161 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4009420 con 0x7f790006c0d0 2024-03-16T19:57:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.043+0000 7f78ff7fe640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c370 tx=0x7f78f000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00052d0 con 0x7f790006c0d0 2024-03-16T19:57:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0005430 con 0x7f790006c0d0 2024-03-16T19:57:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7900142310 con 0x7f790006c0d0 2024-03-16T19:57:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0007260 con 0x7f790006c0d0 2024-03-16T19:57:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79000c3790 con 0x7f790006c0d0 2024-03-16T19:57:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.055+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79000c6230 con 0x7f790006c0d0 2024-03-16T19:57:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.107+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f78f00073c0 con 0x7f790006c0d0 2024-03-16T19:57:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.107+0000 7f78e7fff640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 0x7f78ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.107+0000 7f78feffd640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 0x7f78ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.107+0000 7f78feffd640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 0x7f78ec069350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f78f40b2000 tx=0x7f78f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:33.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.427+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f78f0007670 con 0x7f790006c0d0 2024-03-16T19:57:33.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.459+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f78f00513d0 con 0x7f790006c0d0 2024-03-16T19:57:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.647+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7900068000 con 0x7f790006c0d0 2024-03-16T19:57:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.691+0000 7f78e7fff640 1 -- 172.21.15.32:0/4117151161 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78f0020070 con 0x7f790006c0d0 2024-03-16T19:57:33.663 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 msgr2=0x7f78ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 0x7f78ec069350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f78f40b2000 tx=0x7f78f4008040 comp rx=0 tx=0).stop 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 msgr2=0x7f79000c2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c370 tx=0x7f78f000c830 comp rx=0 tx=0).stop 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 shutdown_connections 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f78ec066e90 0x7f78ec069350 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7900151030 0x7f79001f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79000c7610 0x7f79001f1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 --2- 172.21.15.32:0/4117151161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f790006c0d0 0x7f79000c2b80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 >> 172.21.15.32:0/4117151161 conn(0x7f790006f3d0 msgr2=0x7f79000c8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 shutdown_connections 2024-03-16T19:57:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.695+0000 7f7905ebb640 1 -- 172.21.15.32:0/4117151161 wait complete. 2024-03-16T19:57:33.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T19:57:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 Processor -- start 2024-03-16T19:57:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 -- start start 2024-03-16T19:57:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f409006e710 0x7f409006bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409006c2d0 con 0x7f409006e330 2024-03-16T19:57:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4090072da0 con 0x7f4090157440 2024-03-16T19:57:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f4095f31640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40900c8120 con 0x7f409006e710 2024-03-16T19:57:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f408f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f409006e330 0x7f4090156d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36432/0 (socket says 172.21.15.32:36432) 2024-03-16T19:57:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.819+0000 7f408f7fe640 1 -- 172.21.15.32:0/1429840948 learned_addr learned my addr 172.21.15.32:0/1429840948 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.823+0000 7f408effd640 1 --2- 172.21.15.32:0/1429840948 >> v2:172.21.15.59:3300/0 conn(0x7f409006e710 0x7f409006bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.823+0000 7f408e7fc640 1 -- 172.21.15.32:0/1429840948 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520815937 0 0) 0x7f409006c2d0 con 0x7f409006e330 2024-03-16T19:57:33.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.823+0000 7f408e7fc640 1 -- 172.21.15.32:0/1429840948 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4078003530 con 0x7f409006e330 2024-03-16T19:57:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.827+0000 7f408effd640 1 -- 172.21.15.32:0/1429840948 >> v1:172.21.15.59:6789/0 conn(0x7f4090157440 legacy=0x7f4090150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.827+0000 7f408effd640 1 -- 172.21.15.32:0/1429840948 >> v1:172.21.15.32:6789/0 conn(0x7f409006e330 legacy=0x7f4090156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.827+0000 7f408effd640 1 -- 172.21.15.32:0/1429840948 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40900c8400 con 0x7f409006e710 2024-03-16T19:57:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.835+0000 7f408effd640 1 --2- 172.21.15.32:0/1429840948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f409006e710 0x7f409006bae0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f40840097c0 tx=0x7f408401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80168f8f91fe3ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f408e7fc640 1 -- 172.21.15.32:0/1429840948 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840a8990 con 0x7f409006e710 2024-03-16T19:57:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f408e7fc640 1 -- 172.21.15.32:0/1429840948 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40840a8af0 con 0x7f409006e710 2024-03-16T19:57:33.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f408e7fc640 1 -- 172.21.15.32:0/1429840948 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840a8e80 con 0x7f409006e710 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- 172.21.15.32:0/1429840948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f409006e710 msgr2=0x7f409006bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 --2- 172.21.15.32:0/1429840948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f409006e710 0x7f409006bae0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f40840097c0 tx=0x7f408401ec90 comp rx=0 tx=0).stop 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- 172.21.15.32:0/1429840948 shutdown_connections 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 --2- 172.21.15.32:0/1429840948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f409006e710 0x7f409006bae0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- 172.21.15.32:0/1429840948 >> 172.21.15.32:0/1429840948 conn(0x7f409006afc0 msgr2=0x7f409006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- 172.21.15.32:0/1429840948 shutdown_connections 2024-03-16T19:57:33.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- 172.21.15.32:0/1429840948 wait complete. 2024-03-16T19:57:33.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 Processor -- start 2024-03-16T19:57:33.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.847+0000 7f4095f31640 1 -- start start 2024-03-16T19:57:33.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f409006e710 0x7f40901efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4090157440 0x7f40901f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4090143090 con 0x7f409006e330 2024-03-16T19:57:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409014c2c0 con 0x7f4090157440 2024-03-16T19:57:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f4095f31640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409014b950 con 0x7f409006e710 2024-03-16T19:57:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f408f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f408effd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f409006e710 0x7f40901efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f408f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41942/0 (socket says 172.21.15.32:41942) 2024-03-16T19:57:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f408f7fe640 1 -- 172.21.15.32:0/2450109024 learned_addr learned my addr 172.21.15.32:0/2450109024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.851+0000 7f408ffff640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4090157440 0x7f40901f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.855+0000 7f408f7fe640 1 -- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f409006e710 msgr2=0x7f40901efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.855+0000 7f408f7fe640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f409006e710 0x7f40901efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.855+0000 7f408f7fe640 1 -- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4090157440 msgr2=0x7f40901f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.855+0000 7f408f7fe640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4090157440 0x7f40901f02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.855+0000 7f408f7fe640 1 -- 172.21.15.32:0/2450109024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084009420 con 0x7f409006e330 2024-03-16T19:57:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.859+0000 7f408f7fe640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f408000c480 tx=0x7f408000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.863+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080005cf0 con 0x7f409006e330 2024-03-16T19:57:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.863+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409014be80 con 0x7f409006e330 2024-03-16T19:57:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.863+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409014bba0 con 0x7f409006e330 2024-03-16T19:57:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:33.863+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4090151160 con 0x7f409006e330 2024-03-16T19:57:34.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080005e50 con 0x7f409006e330 2024-03-16T19:57:34.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080007290 con 0x7f409006e330 2024-03-16T19:57:34.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4080007470 con 0x7f409006e330 2024-03-16T19:57:34.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f4077fff640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 0x7f407c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f408effd640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 0x7f407c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.159+0000 7f408effd640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 0x7f407c069300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40840b2000 tx=0x7f40840aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:34.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.183+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4080007720 con 0x7f409006e330 2024-03-16T19:57:34.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.187+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4080051660 con 0x7f409006e330 2024-03-16T19:57:34.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.355+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4090068000 con 0x7f409006e330 2024-03-16T19:57:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.363+0000 7f4077fff640 1 -- 172.21.15.32:0/2450109024 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4080021020 con 0x7f409006e330 2024-03-16T19:57:34.335 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:34.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 msgr2=0x7f407c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 0x7f407c069300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40840b2000 tx=0x7f40840aa070 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 msgr2=0x7f4090142b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f408000c480 tx=0x7f408000c940 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 shutdown_connections 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f407c066e40 0x7f407c069300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4090157440 0x7f40901f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f409006e710 0x7f40901efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 --2- 172.21.15.32:0/2450109024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f409006e330 0x7f4090142b50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 >> 172.21.15.32:0/2450109024 conn(0x7f409006afc0 msgr2=0x7f409014f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 shutdown_connections 2024-03-16T19:57:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.367+0000 7f4095f31640 1 -- 172.21.15.32:0/2450109024 wait complete. 2024-03-16T19:57:34.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:57:34.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 Processor -- start 2024-03-16T19:57:34.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 -- start start 2024-03-16T19:57:34.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64d0151ef0 0x7f64d00c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d015b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d0158c80 con 0x7f64d00c97b0 2024-03-16T19:57:34.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d00b0ad0 con 0x7f64d0151ef0 2024-03-16T19:57:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d7ff8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d0157cb0 con 0x7f64d015b2a0 2024-03-16T19:57:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d5d94640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64d0151ef0 0x7f64d00c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d6595640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d015b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d5d94640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64d0151ef0 0x7f64d00c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41946/0 (socket says 172.21.15.32:41946) 2024-03-16T19:57:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.495+0000 7f64d5d94640 1 -- 172.21.15.32:0/2216897227 learned_addr learned my addr 172.21.15.32:0/2216897227 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.499+0000 7f64d5d94640 1 -- 172.21.15.32:0/2216897227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 msgr2=0x7f64d015b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.499+0000 7f64d5d94640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d015b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.499+0000 7f64d5d94640 1 -- 172.21.15.32:0/2216897227 >> v1:172.21.15.59:6789/0 conn(0x7f64d00c97b0 legacy=0x7f64d00c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.499+0000 7f64d5d94640 1 -- 172.21.15.32:0/2216897227 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d015bfc0 con 0x7f64d0151ef0 2024-03-16T19:57:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.499+0000 7f64d6595640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:57:34.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.503+0000 7f64d5d94640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d00c9270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009870 tx=0x7f64cc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ea678854be340b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:34.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.507+0000 7f64d4d92640 1 -- 172.21.15.32:0/2216897227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a8a50 con 0x7f64d0151ef0 2024-03-16T19:57:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d4d92640 1 -- 172.21.15.32:0/2216897227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc0a8bb0 con 0x7f64d0151ef0 2024-03-16T19:57:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d4d92640 1 -- 172.21.15.32:0/2216897227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0b0440 con 0x7f64d0151ef0 2024-03-16T19:57:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- 172.21.15.32:0/2216897227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 msgr2=0x7f64d00c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d00c9270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009870 tx=0x7f64cc01ed50 comp rx=0 tx=0).stop 2024-03-16T19:57:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- 172.21.15.32:0/2216897227 shutdown_connections 2024-03-16T19:57:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 --2- 172.21.15.32:0/2216897227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d00c9270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- 172.21.15.32:0/2216897227 >> 172.21.15.32:0/2216897227 conn(0x7f64d0070f90 msgr2=0x7f64d00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- 172.21.15.32:0/2216897227 shutdown_connections 2024-03-16T19:57:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- 172.21.15.32:0/2216897227 wait complete. 2024-03-16T19:57:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 Processor -- start 2024-03-16T19:57:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.511+0000 7f64d7ff8640 1 -- start start 2024-03-16T19:57:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64d00c97b0 0x7f64d00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d01ed950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d00b3830 con 0x7f64d0151ef0 2024-03-16T19:57:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d014e200 con 0x7f64d00c97b0 2024-03-16T19:57:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d7ff8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d014df20 con 0x7f64d015b2a0 2024-03-16T19:57:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d5593640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d6595640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d01ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d5593640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41952/0 (socket says 172.21.15.32:41952) 2024-03-16T19:57:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.515+0000 7f64d5593640 1 -- 172.21.15.32:0/4181105106 learned_addr learned my addr 172.21.15.32:0/4181105106 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:34.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 -- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 msgr2=0x7f64d01ed950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d01ed950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 -- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64d00c97b0 msgr2=0x7f64d00b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:34.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64d00c97b0 0x7f64d00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:34.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 -- 172.21.15.32:0/4181105106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8009a70 con 0x7f64d0151ef0 2024-03-16T19:57:34.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.519+0000 7f64d5593640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f64c000c390 tx=0x7f64c000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.523+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c00052d0 con 0x7f64d0151ef0 2024-03-16T19:57:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.523+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c0005430 con 0x7f64d0151ef0 2024-03-16T19:57:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.523+0000 7f64d7ff8640 1 -- 172.21.15.32:0/4181105106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc0094d0 con 0x7f64d0151ef0 2024-03-16T19:57:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.527+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c0003820 con 0x7f64d0151ef0 2024-03-16T19:57:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.527+0000 7f64d7ff8640 1 -- 172.21.15.32:0/4181105106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d00b31e0 con 0x7f64d0151ef0 2024-03-16T19:57:34.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.527+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac0052a0 con 0x7f64d0151ef0 2024-03-16T19:57:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.839+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f64c0003980 con 0x7f64d0151ef0 2024-03-16T19:57:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.839+0000 7f64c67fc640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 0x7f64b4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.839+0000 7f64d5d94640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 0x7f64b4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:34.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.839+0000 7f64d5d94640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 0x7f64b4069350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0b2000 tx=0x7f64cc004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:34.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.879+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f64c00555b0 con 0x7f64d0151ef0 2024-03-16T19:57:34.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:34.907+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f64c0051150 con 0x7f64d0151ef0 2024-03-16T19:57:35.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.183+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64ac005900 con 0x7f64d0151ef0 2024-03-16T19:57:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.255+0000 7f64c67fc640 1 -- 172.21.15.32:0/4181105106 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6637 (secure 0 0 0) 0x7f64ac005900 con 0x7f64d0151ef0 2024-03-16T19:57:35.229 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:35.229 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:57:25.238491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4246,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4246":{"gid":4246,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":16,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4246,4252]},"id":1}]} 2024-03-16T19:57:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-03-16T19:57:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 msgr2=0x7f64b4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 0x7f64b4069350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0b2000 tx=0x7f64cc004740 comp rx=0 tx=0).stop 2024-03-16T19:57:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 msgr2=0x7f64d014d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f64c000c390 tx=0x7f64c000c790 comp rx=0 tx=0).stop 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 shutdown_connections 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f64b4066e90 0x7f64b4069350 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f64d015b2a0 0x7f64d01ed950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64d0151ef0 0x7f64d014d870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 --2- 172.21.15.32:0/4181105106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64d00c97b0 0x7f64d00b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 >> 172.21.15.32:0/4181105106 conn(0x7f64d0070f90 msgr2=0x7f64d0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 shutdown_connections 2024-03-16T19:57:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.259+0000 7f64b3fff640 1 -- 172.21.15.32:0/4181105106 wait complete. 2024-03-16T19:57:35.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.379+0000 7f4e94183640 1 Processor -- start 2024-03-16T19:57:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.379+0000 7f4e94183640 1 -- start start 2024-03-16T19:57:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e94183640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e8c0c7230 0x7f4e8c06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e94183640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4e8c158b30 0x7f4e8c162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e94183640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c0c5ce0 con 0x7f4e8c073350 2024-03-16T19:57:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e94183640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c06afc0 con 0x7f4e8c0c7230 2024-03-16T19:57:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e94183640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c072f80 con 0x7f4e8c158b30 2024-03-16T19:57:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e91f1f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e8c0c7230 0x7f4e8c06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e92720640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4e8c158b30 0x7f4e8c162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e91f1f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e8c0c7230 0x7f4e8c06b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41968/0 (socket says 172.21.15.32:41968) 2024-03-16T19:57:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.383+0000 7f4e91f1f640 1 -- 172.21.15.32:0/2100846472 learned_addr learned my addr 172.21.15.32:0/2100846472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:35.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.387+0000 7f4e91f1f640 1 -- 172.21.15.32:0/2100846472 >> v2:172.21.15.59:3300/0 conn(0x7f4e8c158b30 msgr2=0x7f4e8c162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.387+0000 7f4e91f1f640 1 --2- 172.21.15.32:0/2100846472 >> v2:172.21.15.59:3300/0 conn(0x7f4e8c158b30 0x7f4e8c162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.387+0000 7f4e91f1f640 1 -- 172.21.15.32:0/2100846472 >> v1:172.21.15.59:6789/0 conn(0x7f4e8c073350 legacy=0x7f4e8c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.387+0000 7f4e91f1f640 1 -- 172.21.15.32:0/2100846472 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0c5f30 con 0x7f4e8c0c7230 2024-03-16T19:57:35.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.391+0000 7f4e91f1f640 1 --2- 172.21.15.32:0/2100846472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c06b450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0c8210 tx=0x7f4e8801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917a676a304f8368 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e90f1d640 1 -- 172.21.15.32:0/2100846472 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880a8a10 con 0x7f4e8c0c7230 2024-03-16T19:57:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e90f1d640 1 -- 172.21.15.32:0/2100846472 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880a8b70 con 0x7f4e8c0c7230 2024-03-16T19:57:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e90f1d640 1 -- 172.21.15.32:0/2100846472 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880b0440 con 0x7f4e8c0c7230 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 -- 172.21.15.32:0/2100846472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c0c7230 msgr2=0x7f4e8c06b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 --2- 172.21.15.32:0/2100846472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c06b450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0c8210 tx=0x7f4e8801ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 -- 172.21.15.32:0/2100846472 shutdown_connections 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 --2- 172.21.15.32:0/2100846472 >> v2:172.21.15.59:3300/0 conn(0x7f4e8c158b30 0x7f4e8c162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 --2- 172.21.15.32:0/2100846472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c06b450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 -- 172.21.15.32:0/2100846472 >> 172.21.15.32:0/2100846472 conn(0x7f4e8c0cc7a0 msgr2=0x7f4e8c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 -- 172.21.15.32:0/2100846472 shutdown_connections 2024-03-16T19:57:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.395+0000 7f4e94183640 1 -- 172.21.15.32:0/2100846472 wait complete. 2024-03-16T19:57:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 Processor -- start 2024-03-16T19:57:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 -- start start 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c13f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4e8c158b30 0x7f4e8c1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c13f1d0 con 0x7f4e8c073350 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c1f0750 con 0x7f4e8c0c7230 2024-03-16T19:57:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e94183640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c1526d0 con 0x7f4e8c158b30 2024-03-16T19:57:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e91f1f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e92720640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4e8c158b30 0x7f4e8c1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e91f1f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41978/0 (socket says 172.21.15.32:41978) 2024-03-16T19:57:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e91f1f640 1 -- 172.21.15.32:0/99214414 learned_addr learned my addr 172.21.15.32:0/99214414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.399+0000 7f4e9171e640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c13f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.403+0000 7f4e91f1f640 1 -- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4e8c158b30 msgr2=0x7f4e8c1f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.403+0000 7f4e91f1f640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4e8c158b30 0x7f4e8c1f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.403+0000 7f4e91f1f640 1 -- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e8c0c7230 msgr2=0x7f4e8c13f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.403+0000 7f4e91f1f640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c13f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.407+0000 7f4e91f1f640 1 -- 172.21.15.32:0/99214414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880094d0 con 0x7f4e8c073350 2024-03-16T19:57:35.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.407+0000 7f4e91f1f640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4e8801b920 tx=0x7f4e8801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:35.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880ba600 con 0x7f4e8c073350 2024-03-16T19:57:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880ba760 con 0x7f4e8c073350 2024-03-16T19:57:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c1529a0 con 0x7f4e8c073350 2024-03-16T19:57:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c152f60 con 0x7f4e8c073350 2024-03-16T19:57:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880bab10 con 0x7f4e8c073350 2024-03-16T19:57:35.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.415+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e740052a0 con 0x7f4e8c073350 2024-03-16T19:57:35.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.699+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4e880bac70 con 0x7f4e8c073350 2024-03-16T19:57:35.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.699+0000 7f4e827fc640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 0x7f4e70069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:35.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.699+0000 7f4e9171e640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 0x7f4e70069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:35.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.699+0000 7f4e9171e640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 0x7f4e70069350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0c5cb0 tx=0x7f4e7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:35.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.719+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4e880f4690 con 0x7f4e8c073350 2024-03-16T19:57:35.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.719+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4e880f0930 con 0x7f4e8c073350 2024-03-16T19:57:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.883+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e74005820 con 0x7f4e8c073350 2024-03-16T19:57:35.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.939+0000 7f4e827fc640 1 -- 172.21.15.32:0/99214414 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f4e880b5030 con 0x7f4e8c073350 2024-03-16T19:57:35.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:35.912 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:56:44.487941+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 msgr2=0x7f4e70069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 0x7f4e70069350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0c5cb0 tx=0x7f4e7c009040 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 msgr2=0x7f4e8c153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4e8801b920 tx=0x7f4e8801bab0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 shutdown_connections 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4e70066e90 0x7f4e70069350 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4e8c158b30 0x7f4e8c1f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e8c0c7230 0x7f4e8c13f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 --2- 172.21.15.32:0/99214414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c073350 0x7f4e8c153290 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 >> 172.21.15.32:0/99214414 conn(0x7f4e8c0cc7a0 msgr2=0x7f4e8c06bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 shutdown_connections 2024-03-16T19:57:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:35.943+0000 7f4e94183640 1 -- 172.21.15.32:0/99214414 wait complete. 2024-03-16T19:57:35.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:55:53.065249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:56:31.637719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:56:33.839961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T19:56:40.663264+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T19:57:35.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:57:36.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.071+0000 7f8c738c4640 1 Processor -- start 2024-03-16T19:57:36.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.071+0000 7f8c738c4640 1 -- start start 2024-03-16T19:57:36.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.075+0000 7f8c738c4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8c6c154030 0x7f8c6c070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.075+0000 7f8c738c4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c159f80 con 0x7f8c6c0708a0 2024-03-16T19:57:36.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.075+0000 7f8c738c4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c073600 con 0x7f8c6c15ab20 2024-03-16T19:57:36.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.075+0000 7f8c738c4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c15d470 con 0x7f8c6c154030 2024-03-16T19:57:36.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.079+0000 7f8c70e5f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8c6c0708a0 0x7f8c6c15a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36434/0 (socket says 172.21.15.32:36434) 2024-03-16T19:57:36.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.079+0000 7f8c70e5f640 1 -- 172.21.15.32:0/451998860 learned_addr learned my addr 172.21.15.32:0/451998860 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.083+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130246121 0 0) 0x7f8c6c159f80 con 0x7f8c6c0708a0 2024-03-16T19:57:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.083+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58003530 con 0x7f8c6c0708a0 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097865045 0 0) 0x7f8c58003530 con 0x7f8c6c0708a0 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 >> v2:172.21.15.59:3300/0 conn(0x7f8c6c154030 msgr2=0x7f8c6c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 --2- 172.21.15.32:0/451998860 >> v2:172.21.15.59:3300/0 conn(0x7f8c6c154030 0x7f8c6c070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 >> v1:172.21.15.59:6789/0 conn(0x7f8c6c15ab20 legacy=0x7f8c6c0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c15d710 con 0x7f8c6c0708a0 2024-03-16T19:57:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f8c5c001680 con 0x7f8c6c0708a0 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 >> v1:172.21.15.32:6789/0 conn(0x7f8c6c0708a0 legacy=0x7f8c6c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c58003980 0x7f8c58003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c580047d0 0x7f8c58004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 0x7f8c5800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c15d470 con 0x7f8c580052b0 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5800c1b0 con 0x7f8c580047d0 2024-03-16T19:57:36.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5800c370 con 0x7f8c58003980 2024-03-16T19:57:36.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.087+0000 7f8c71e61640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 0x7f8c5800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.091+0000 7f8c71e61640 1 -- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c58003980 msgr2=0x7f8c58003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:36.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.091+0000 7f8c71e61640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c58003980 0x7f8c58003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.091+0000 7f8c71e61640 1 -- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c580047d0 msgr2=0x7f8c58004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:36.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.091+0000 7f8c71e61640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c580047d0 0x7f8c58004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.091+0000 7f8c71e61640 1 -- 172.21.15.32:0/451998860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c15d710 con 0x7f8c580052b0 2024-03-16T19:57:36.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.095+0000 7f8c71e61640 1 -- 172.21.15.32:0/451998860 reap_dead start 2024-03-16T19:57:36.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.099+0000 7f8c71e61640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 0x7f8c5800b990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8c64007fd0 tx=0x7f8c6401c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db95f7bb372d075 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.107+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64003ba0 con 0x7f8c580052b0 2024-03-16T19:57:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.107+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64003d20 con 0x7f8c580052b0 2024-03-16T19:57:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.107+0000 7f8c63fff640 1 -- 172.21.15.32:0/451998860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640052b0 con 0x7f8c580052b0 2024-03-16T19:57:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.107+0000 7f8c738c4640 1 -- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 msgr2=0x7f8c5800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.107+0000 7f8c738c4640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 0x7f8c5800b990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8c64007fd0 tx=0x7f8c6401c680 comp rx=0 tx=0).stop 2024-03-16T19:57:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- 172.21.15.32:0/451998860 shutdown_connections 2024-03-16T19:57:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 --2- 172.21.15.32:0/451998860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580052b0 0x7f8c5800b990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- 172.21.15.32:0/451998860 >> 172.21.15.32:0/451998860 conn(0x7f8c6c06c0d0 msgr2=0x7f8c6c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- 172.21.15.32:0/451998860 shutdown_connections 2024-03-16T19:57:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- 172.21.15.32:0/451998860 wait complete. 2024-03-16T19:57:36.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 Processor -- start 2024-03-16T19:57:36.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- start start 2024-03-16T19:57:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c580052b0 0x7f8c6c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c6c15b040 0x7f8c6c13c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c50a0 con 0x7f8c6c159a10 2024-03-16T19:57:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c4a20 con 0x7f8c6c15b040 2024-03-16T19:57:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c738c4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c5a00 con 0x7f8c580052b0 2024-03-16T19:57:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.111+0000 7f8c70e5f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c71660640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c580052b0 0x7f8c6c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c71e61640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c6c15b040 0x7f8c6c13c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41990/0 (socket says 172.21.15.32:41990) 2024-03-16T19:57:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 -- 172.21.15.32:0/3144047293 learned_addr learned my addr 172.21.15.32:0/3144047293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 -- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c580052b0 msgr2=0x7f8c6c0c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c580052b0 0x7f8c6c0c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 -- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c6c15b040 msgr2=0x7f8c6c13c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c6c15b040 0x7f8c6c13c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.115+0000 7f8c70e5f640 1 -- 172.21.15.32:0/3144047293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c1584e0 con 0x7f8c6c159a10 2024-03-16T19:57:36.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.119+0000 7f8c70e5f640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00c780 tx=0x7f8c5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0074e0 con 0x7f8c6c159a10 2024-03-16T19:57:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c007680 con 0x7f8c6c159a10 2024-03-16T19:57:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c1587c0 con 0x7f8c6c159a10 2024-03-16T19:57:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c004a30 con 0x7f8c6c159a10 2024-03-16T19:57:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c0c4cc0 con 0x7f8c6c159a10 2024-03-16T19:57:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.123+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c0c4e80 con 0x7f8c6c159a10 2024-03-16T19:57:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.391+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c5c004b90 con 0x7f8c6c159a10 2024-03-16T19:57:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.391+0000 7f8c61ffb640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 0x7f8c50069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.391+0000 7f8c71660640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 0x7f8c50069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.391+0000 7f8c71660640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 0x7f8c50069300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006fd0 tx=0x7f8c680090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.415+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8c5c00ce30 con 0x7f8c6c159a10 2024-03-16T19:57:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.415+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8c5c051510 con 0x7f8c6c159a10 2024-03-16T19:57:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.587+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c6c0c51e0 con 0x7f8c6c159a10 2024-03-16T19:57:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.619+0000 7f8c61ffb640 1 -- 172.21.15.32:0/3144047293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c5c018030 con 0x7f8c6c159a10 2024-03-16T19:57:36.592 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-16T19:57:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 msgr2=0x7f8c50069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 0x7f8c50069300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006fd0 tx=0x7f8c680090e0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 msgr2=0x7f8c6c157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00c780 tx=0x7f8c5c00cc40 comp rx=0 tx=0).stop 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 shutdown_connections 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c50066e40 0x7f8c50069300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c6c15b040 0x7f8c6c13c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6c159a10 0x7f8c6c157ca0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 --2- 172.21.15.32:0/3144047293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c580052b0 0x7f8c6c0c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 >> 172.21.15.32:0/3144047293 conn(0x7f8c6c06c0d0 msgr2=0x7f8c6c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 shutdown_connections 2024-03-16T19:57:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.623+0000 7f8c738c4640 1 -- 172.21.15.32:0/3144047293 wait complete. 2024-03-16T19:57:36.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:57:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.743+0000 7f6a19c0f640 1 Processor -- start 2024-03-16T19:57:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.743+0000 7f6a19c0f640 1 -- start start 2024-03-16T19:57:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a19c0f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a14151090 0x7f6a14072a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a19c0f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a140c89f0 0x7f6a14157020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a19c0f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a140c5e60 con 0x7f6a14073140 2024-03-16T19:57:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a19c0f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a14156bd0 con 0x7f6a14151090 2024-03-16T19:57:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a19c0f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a140c6140 con 0x7f6a140c89f0 2024-03-16T19:57:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a137fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a14151090 0x7f6a14072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a13fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a140c89f0 0x7f6a14157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a137fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a14151090 0x7f6a14072a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42006/0 (socket says 172.21.15.32:42006) 2024-03-16T19:57:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.747+0000 7f6a137fe640 1 -- 172.21.15.32:0/3736696355 learned_addr learned my addr 172.21.15.32:0/3736696355 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.751+0000 7f6a137fe640 1 -- 172.21.15.32:0/3736696355 >> v2:172.21.15.59:3300/0 conn(0x7f6a140c89f0 msgr2=0x7f6a14157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.751+0000 7f6a137fe640 1 --2- 172.21.15.32:0/3736696355 >> v2:172.21.15.59:3300/0 conn(0x7f6a140c89f0 0x7f6a14157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.751+0000 7f6a137fe640 1 -- 172.21.15.32:0/3736696355 >> v1:172.21.15.59:6789/0 conn(0x7f6a14073140 legacy=0x7f6a1406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.751+0000 7f6a137fe640 1 -- 172.21.15.32:0/3736696355 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140c6420 con 0x7f6a14151090 2024-03-16T19:57:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.755+0000 7f6a137fe640 1 --2- 172.21.15.32:0/3736696355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a14151090 0x7f6a14072a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6a14070b30 tx=0x7f6a040213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb19cbfcc9167ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.759+0000 7f6a127fc640 1 -- 172.21.15.32:0/3736696355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a040b1600 con 0x7f6a14151090 2024-03-16T19:57:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.759+0000 7f6a127fc640 1 -- 172.21.15.32:0/3736696355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a040ad050 con 0x7f6a14151090 2024-03-16T19:57:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.759+0000 7f6a127fc640 1 -- 172.21.15.32:0/3736696355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a040b1980 con 0x7f6a14151090 2024-03-16T19:57:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- 172.21.15.32:0/3736696355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a14151090 msgr2=0x7f6a14072a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 --2- 172.21.15.32:0/3736696355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a14151090 0x7f6a14072a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6a14070b30 tx=0x7f6a040213e0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- 172.21.15.32:0/3736696355 shutdown_connections 2024-03-16T19:57:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 --2- 172.21.15.32:0/3736696355 >> v2:172.21.15.59:3300/0 conn(0x7f6a140c89f0 0x7f6a14157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 --2- 172.21.15.32:0/3736696355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a14151090 0x7f6a14072a20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- 172.21.15.32:0/3736696355 >> 172.21.15.32:0/3736696355 conn(0x7f6a140cc7a0 msgr2=0x7f6a140ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- 172.21.15.32:0/3736696355 shutdown_connections 2024-03-16T19:57:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- 172.21.15.32:0/3736696355 wait complete. 2024-03-16T19:57:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 Processor -- start 2024-03-16T19:57:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.763+0000 7f6a19c0f640 1 -- start start 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14073140 0x7f6a1414f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a14151090 0x7f6a141f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141f0630 con 0x7f6a140c89f0 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141f4060 con 0x7f6a14073140 2024-03-16T19:57:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a19c0f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141f4340 con 0x7f6a14151090 2024-03-16T19:57:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a12ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a12ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42010/0 (socket says 172.21.15.32:42010) 2024-03-16T19:57:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a12ffd640 1 -- 172.21.15.32:0/1521981877 learned_addr learned my addr 172.21.15.32:0/1521981877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a137fe640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14073140 0x7f6a1414f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.767+0000 7f6a13fff640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a14151090 0x7f6a141f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 -- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a14151090 msgr2=0x7f6a141f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a14151090 0x7f6a141f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 -- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14073140 msgr2=0x7f6a1414f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14073140 0x7f6a1414f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 -- 172.21.15.32:0/1521981877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0400b770 con 0x7f6a140c89f0 2024-03-16T19:57:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.771+0000 7f6a12ffd640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c420 tx=0x7f6a0800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:36.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.775+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a080052d0 con 0x7f6a140c89f0 2024-03-16T19:57:36.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.775+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a08005430 con 0x7f6a140c89f0 2024-03-16T19:57:36.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.775+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08007260 con 0x7f6a140c89f0 2024-03-16T19:57:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.779+0000 7f6a19c0f640 1 -- 172.21.15.32:0/1521981877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a141f4620 con 0x7f6a140c89f0 2024-03-16T19:57:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.779+0000 7f6a19c0f640 1 -- 172.21.15.32:0/1521981877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a141f4900 con 0x7f6a140c89f0 2024-03-16T19:57:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:36.779+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec0052a0 con 0x7f6a140c89f0 2024-03-16T19:57:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.071+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6a080073c0 con 0x7f6a140c89f0 2024-03-16T19:57:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.071+0000 7f69fbfff640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 0x7f6a00069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.071+0000 7f6a137fe640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 0x7f6a00069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.071+0000 7f6a137fe640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 0x7f6a00069300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6a040b3000 tx=0x7f6a040ac000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:37.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.115+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6a08007670 con 0x7f6a140c89f0 2024-03-16T19:57:37.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.119+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6a08051550 con 0x7f6a140c89f0 2024-03-16T19:57:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.379+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69ec005ad0 con 0x7f6a140c89f0 2024-03-16T19:57:37.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69fbfff640 1 -- 172.21.15.32:0/1521981877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a08020020 con 0x7f6a140c89f0 2024-03-16T19:57:37.442 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 msgr2=0x7f6a00069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 0x7f6a00069300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6a040b3000 tx=0x7f6a040ac000 comp rx=0 tx=0).stop 2024-03-16T19:57:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 msgr2=0x7f6a1415a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c420 tx=0x7f6a0800c8e0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 shutdown_connections 2024-03-16T19:57:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a00066e40 0x7f6a00069300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a14151090 0x7f6a141f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a140c89f0 0x7f6a1415a3e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 --2- 172.21.15.32:0/1521981877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14073140 0x7f6a1414f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 >> 172.21.15.32:0/1521981877 conn(0x7f6a140cc7a0 msgr2=0x7f6a140c77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:37.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 shutdown_connections 2024-03-16T19:57:37.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.471+0000 7f69f9ffb640 1 -- 172.21.15.32:0/1521981877 wait complete. 2024-03-16T19:57:37.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:57:37.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 Processor -- start 2024-03-16T19:57:37.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 -- start start 2024-03-16T19:57:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2318151e50 0x7f23180c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23180c97b0 0x7f2318157d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231815bdf0 con 0x7f231806f0a0 2024-03-16T19:57:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2318158d30 con 0x7f2318151e50 2024-03-16T19:57:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231e406640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2318152230 con 0x7f23180c97b0 2024-03-16T19:57:37.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f2317fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2318151e50 0x7f23180c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f2317fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2318151e50 0x7f23180c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42014/0 (socket says 172.21.15.32:42014) 2024-03-16T19:57:37.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f2317fff640 1 -- 172.21.15.32:0/1900582968 learned_addr learned my addr 172.21.15.32:0/1900582968 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:37.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.603+0000 7f231c9a3640 1 --2- 172.21.15.32:0/1900582968 >> v2:172.21.15.59:3300/0 conn(0x7f23180c97b0 0x7f2318157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.607+0000 7f2317fff640 1 -- 172.21.15.32:0/1900582968 >> v2:172.21.15.59:3300/0 conn(0x7f23180c97b0 msgr2=0x7f2318157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.607+0000 7f2317fff640 1 --2- 172.21.15.32:0/1900582968 >> v2:172.21.15.59:3300/0 conn(0x7f23180c97b0 0x7f2318157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.607+0000 7f2317fff640 1 -- 172.21.15.32:0/1900582968 >> v1:172.21.15.59:6789/0 conn(0x7f231806f0a0 legacy=0x7f23180c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.607+0000 7f2317fff640 1 -- 172.21.15.32:0/1900582968 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231815c280 con 0x7f2318151e50 2024-03-16T19:57:37.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.611+0000 7f2317fff640 1 --2- 172.21.15.32:0/1900582968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318151e50 0x7f23180c9270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f23180706b0 tx=0x7f230801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7154305dd717d1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f2316ffd640 1 -- 172.21.15.32:0/1900582968 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080a8a10 con 0x7f2318151e50 2024-03-16T19:57:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f2316ffd640 1 -- 172.21.15.32:0/1900582968 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23080a8b70 con 0x7f2318151e50 2024-03-16T19:57:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f2316ffd640 1 -- 172.21.15.32:0/1900582968 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080b0440 con 0x7f2318151e50 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- 172.21.15.32:0/1900582968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318151e50 msgr2=0x7f23180c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 --2- 172.21.15.32:0/1900582968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318151e50 0x7f23180c9270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f23180706b0 tx=0x7f230801ed10 comp rx=0 tx=0).stop 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- 172.21.15.32:0/1900582968 shutdown_connections 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 --2- 172.21.15.32:0/1900582968 >> v2:172.21.15.59:3300/0 conn(0x7f23180c97b0 0x7f2318157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 --2- 172.21.15.32:0/1900582968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318151e50 0x7f23180c9270 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- 172.21.15.32:0/1900582968 >> 172.21.15.32:0/1900582968 conn(0x7f2318070f90 msgr2=0x7f23180713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- 172.21.15.32:0/1900582968 shutdown_connections 2024-03-16T19:57:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- 172.21.15.32:0/1900582968 wait complete. 2024-03-16T19:57:37.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 Processor -- start 2024-03-16T19:57:37.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.615+0000 7f231e406640 1 -- start start 2024-03-16T19:57:37.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23180c97b0 0x7f231814b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2318151e50 0x7f231814b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180cbb50 con 0x7f231806f0a0 2024-03-16T19:57:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23181f1b50 con 0x7f23180c97b0 2024-03-16T19:57:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231e406640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23180cdec0 con 0x7f2318151e50 2024-03-16T19:57:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f2317fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f2317fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42028/0 (socket says 172.21.15.32:42028) 2024-03-16T19:57:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f2317fff640 1 -- 172.21.15.32:0/1298391282 learned_addr learned my addr 172.21.15.32:0/1298391282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f23177fe640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23180c97b0 0x7f231814b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.619+0000 7f231c9a3640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2318151e50 0x7f231814b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 -- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2318151e50 msgr2=0x7f231814b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2318151e50 0x7f231814b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 -- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23180c97b0 msgr2=0x7f231814b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23180c97b0 0x7f231814b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 -- 172.21.15.32:0/1298391282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080094d0 con 0x7f231806f0a0 2024-03-16T19:57:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.623+0000 7f2317fff640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f230801b920 tx=0x7f230801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:37.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080ba600 con 0x7f231806f0a0 2024-03-16T19:57:37.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23080ba760 con 0x7f231806f0a0 2024-03-16T19:57:37.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080bab10 con 0x7f231806f0a0 2024-03-16T19:57:37.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23181f1870 con 0x7f231806f0a0 2024-03-16T19:57:37.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23180cdae0 con 0x7f231806f0a0 2024-03-16T19:57:37.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.631+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318068000 con 0x7f231806f0a0 2024-03-16T19:57:37.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.911+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f23080bac70 con 0x7f231806f0a0 2024-03-16T19:57:37.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.911+0000 7f2314ff9640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 0x7f2304069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:37.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.911+0000 7f23177fe640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 0x7f2304069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:37.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.911+0000 7f23177fe640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 0x7f2304069300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f23180b0730 tx=0x7f230c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.931+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f23080f45a0 con 0x7f231806f0a0 2024-03-16T19:57:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:37.931+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f23080f0840 con 0x7f231806f0a0 2024-03-16T19:57:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.087+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f231814ecd0 con 0x7f231806f0a0 2024-03-16T19:57:38.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f2314ff9640 1 -- 172.21.15.32:0/1298391282 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f23080b5070 con 0x7f231806f0a0 2024-03-16T19:57:38.070 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-16T19:57:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 msgr2=0x7f2304069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 0x7f2304069300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f23180b0730 tx=0x7f230c009040 comp rx=0 tx=0).stop 2024-03-16T19:57:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 msgr2=0x7f23181f0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f230801b920 tx=0x7f230801bab0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 shutdown_connections 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2304066e40 0x7f2304069300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2318151e50 0x7f231814b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23180c97b0 0x7f231814b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 --2- 172.21.15.32:0/1298391282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231806f0a0 0x7f23181f0ea0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 >> 172.21.15.32:0/1298391282 conn(0x7f2318070f90 msgr2=0x7f2318150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 shutdown_connections 2024-03-16T19:57:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.099+0000 7f231e406640 1 -- 172.21.15.32:0/1298391282 wait complete. 2024-03-16T19:57:38.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T19:57:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 Processor -- start 2024-03-16T19:57:38.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 -- start start 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36b40b1920 0x7f36b40c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f36b40b1d00 0x7f36b4066ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b406c9b0 con 0x7f36b40c6920 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b4067410 con 0x7f36b40b1d00 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36baf6b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b40c7b10 con 0x7f36b40b1920 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36b3fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f36b40b1d00 0x7f36b4066ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36b3fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f36b40b1d00 0x7f36b4066ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42038/0 (socket says 172.21.15.32:42038) 2024-03-16T19:57:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36b3fff640 1 -- 172.21.15.32:0/2477845218 learned_addr learned my addr 172.21.15.32:0/2477845218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.223+0000 7f36b8d07640 1 --2- 172.21.15.32:0/2477845218 >> v2:172.21.15.59:3300/0 conn(0x7f36b40b1920 0x7f36b40c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.227+0000 7f36b3fff640 1 -- 172.21.15.32:0/2477845218 >> v2:172.21.15.59:3300/0 conn(0x7f36b40b1920 msgr2=0x7f36b40c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.227+0000 7f36b3fff640 1 --2- 172.21.15.32:0/2477845218 >> v2:172.21.15.59:3300/0 conn(0x7f36b40b1920 0x7f36b40c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.227+0000 7f36b3fff640 1 -- 172.21.15.32:0/2477845218 >> v1:172.21.15.59:6789/0 conn(0x7f36b40c6920 legacy=0x7f36b40c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.227+0000 7f36b3fff640 1 -- 172.21.15.32:0/2477845218 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40c84c0 con 0x7f36b40b1d00 2024-03-16T19:57:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.231+0000 7f36b3fff640 1 --2- 172.21.15.32:0/2477845218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1d00 0x7f36b4066ed0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f36b4073480 tx=0x7f36a001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0bb8c378f1e1b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36b37fe640 1 -- 172.21.15.32:0/2477845218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00aa640 con 0x7f36b40b1d00 2024-03-16T19:57:38.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36b37fe640 1 -- 172.21.15.32:0/2477845218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a00aa7a0 con 0x7f36b40b1d00 2024-03-16T19:57:38.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36b37fe640 1 -- 172.21.15.32:0/2477845218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00aab30 con 0x7f36b40b1d00 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 -- 172.21.15.32:0/2477845218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1d00 msgr2=0x7f36b4066ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 --2- 172.21.15.32:0/2477845218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1d00 0x7f36b4066ed0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f36b4073480 tx=0x7f36a001c4e0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 -- 172.21.15.32:0/2477845218 shutdown_connections 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 --2- 172.21.15.32:0/2477845218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1d00 0x7f36b4066ed0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 --2- 172.21.15.32:0/2477845218 >> v2:172.21.15.59:3300/0 conn(0x7f36b40b1920 0x7f36b40c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 -- 172.21.15.32:0/2477845218 >> 172.21.15.32:0/2477845218 conn(0x7f36b406f3a0 msgr2=0x7f36b406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 -- 172.21.15.32:0/2477845218 shutdown_connections 2024-03-16T19:57:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 -- 172.21.15.32:0/2477845218 wait complete. 2024-03-16T19:57:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.235+0000 7f36baf6b640 1 Processor -- start 2024-03-16T19:57:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 -- start start 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36b40b1d00 0x7f36b41618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36b40c6920 0x7f36b406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41441a0 con 0x7f36b40b1920 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4148ee0 con 0x7f36b40c6920 2024-03-16T19:57:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36baf6b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41f3100 con 0x7f36b40b1d00 2024-03-16T19:57:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36b8d07640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36b9508640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36b40c6920 0x7f36b406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36b8d07640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42054/0 (socket says 172.21.15.32:42054) 2024-03-16T19:57:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36b8d07640 1 -- 172.21.15.32:0/3582807145 learned_addr learned my addr 172.21.15.32:0/3582807145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.239+0000 7f36b3fff640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36b40b1d00 0x7f36b41618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.243+0000 7f36b8d07640 1 -- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36b40b1d00 msgr2=0x7f36b41618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.243+0000 7f36b8d07640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36b40b1d00 0x7f36b41618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.243+0000 7f36b8d07640 1 -- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36b40c6920 msgr2=0x7f36b406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.243+0000 7f36b8d07640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36b40c6920 0x7f36b406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.243+0000 7f36b8d07640 1 -- 172.21.15.32:0/3582807145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009480 con 0x7f36b40b1920 2024-03-16T19:57:38.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.247+0000 7f36b8d07640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36a800e930 tx=0x7f36a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.251+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8009690 con 0x7f36b40b1920 2024-03-16T19:57:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.251+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a800c9a0 con 0x7f36b40b1920 2024-03-16T19:57:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.251+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a800cd10 con 0x7f36b40b1920 2024-03-16T19:57:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.251+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b414a0b0 con 0x7f36b40b1920 2024-03-16T19:57:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.251+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b41485b0 con 0x7f36b40b1920 2024-03-16T19:57:38.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.255+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b4068000 con 0x7f36b40b1920 2024-03-16T19:57:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.291+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f36a8013470 con 0x7f36b40b1920 2024-03-16T19:57:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.295+0000 7f36b17fa640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 0x7f3698069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.295+0000 7f36b3fff640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 0x7f3698069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.295+0000 7f36b3fff640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 0x7f3698069300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f36a00ae000 tx=0x7f36a00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.567+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f36a80560a0 con 0x7f36b40b1920 2024-03-16T19:57:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.567+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f36a8052340 con 0x7f36b40b1920 2024-03-16T19:57:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.739+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f36b40c4f00 con 0x7f36b40b1920 2024-03-16T19:57:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36b17fa640 1 -- 172.21.15.32:0/3582807145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f36a80210a0 con 0x7f36b40b1920 2024-03-16T19:57:38.730 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-16T19:57:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 msgr2=0x7f3698069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 0x7f3698069300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f36a00ae000 tx=0x7f36a00a7000 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 msgr2=0x7f36b4143c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36a800e930 tx=0x7f36a800edf0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 shutdown_connections 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3698066e40 0x7f3698069300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36b40c6920 0x7f36b406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36b40b1d00 0x7f36b41618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 --2- 172.21.15.32:0/3582807145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b40b1920 0x7f36b4143c60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 >> 172.21.15.32:0/3582807145 conn(0x7f36b406f3a0 msgr2=0x7f36b4066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 shutdown_connections 2024-03-16T19:57:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.759+0000 7f36baf6b640 1 -- 172.21.15.32:0/3582807145 wait complete. 2024-03-16T19:57:38.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:57:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 Processor -- start 2024-03-16T19:57:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 -- start start 2024-03-16T19:57:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e200c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e20071a50 0x7f6e20071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e201546e0 con 0x7f6e20072520 2024-03-16T19:57:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e20159fb0 con 0x7f6e20071a50 2024-03-16T19:57:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.879+0000 7f6e26319640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200c7400 con 0x7f6e20154300 2024-03-16T19:57:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.883+0000 7f6e248b6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6e20072520 0x7f6e2015a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:45946/0 (socket says 172.21.15.32:45946) 2024-03-16T19:57:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.883+0000 7f6e248b6640 1 -- 172.21.15.32:0/1943348800 learned_addr learned my addr 172.21.15.32:0/1943348800 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.883+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/1943348800 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e200c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1effd640 1 -- 172.21.15.32:0/1943348800 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876498478 0 0) 0x7f6e201546e0 con 0x7f6e20072520 2024-03-16T19:57:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1effd640 1 -- 172.21.15.32:0/1943348800 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003530 con 0x7f6e20072520 2024-03-16T19:57:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1ffff640 1 -- 172.21.15.32:0/1943348800 >> v2:172.21.15.32:3300/0 conn(0x7f6e20071a50 msgr2=0x7f6e20071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/1943348800 >> v2:172.21.15.32:3300/0 conn(0x7f6e20071a50 0x7f6e20071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1ffff640 1 -- 172.21.15.32:0/1943348800 >> v1:172.21.15.59:6789/0 conn(0x7f6e20072520 legacy=0x7f6e2015a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.887+0000 7f6e1ffff640 1 -- 172.21.15.32:0/1943348800 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200c76e0 con 0x7f6e20154300 2024-03-16T19:57:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.895+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/1943348800 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e200c69e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6e100097c0 tx=0x7f6e1001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=660754f2be98567f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.899+0000 7f6e1effd640 1 -- 172.21.15.32:0/1943348800 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e100ad680 con 0x7f6e20154300 2024-03-16T19:57:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.899+0000 7f6e1effd640 1 -- 172.21.15.32:0/1943348800 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e100ad7e0 con 0x7f6e20154300 2024-03-16T19:57:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.899+0000 7f6e1effd640 1 -- 172.21.15.32:0/1943348800 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e100adb70 con 0x7f6e20154300 2024-03-16T19:57:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- 172.21.15.32:0/1943348800 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 msgr2=0x7f6e200c69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 --2- 172.21.15.32:0/1943348800 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e200c69e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6e100097c0 tx=0x7f6e1001c470 comp rx=0 tx=0).stop 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- 172.21.15.32:0/1943348800 shutdown_connections 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 --2- 172.21.15.32:0/1943348800 >> v2:172.21.15.32:3300/0 conn(0x7f6e20071a50 0x7f6e20071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 --2- 172.21.15.32:0/1943348800 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e200c69e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- 172.21.15.32:0/1943348800 >> 172.21.15.32:0/1943348800 conn(0x7f6e200cbfe0 msgr2=0x7f6e200cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- 172.21.15.32:0/1943348800 shutdown_connections 2024-03-16T19:57:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- 172.21.15.32:0/1943348800 wait complete. 2024-03-16T19:57:38.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 Processor -- start 2024-03-16T19:57:38.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.903+0000 7f6e26319640 1 -- start start 2024-03-16T19:57:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e20071a50 0x7f6e20148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e201f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2015ce70 con 0x7f6e20072520 2024-03-16T19:57:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e201f2f60 con 0x7f6e20071a50 2024-03-16T19:57:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e26319640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e201f6670 con 0x7f6e20154300 2024-03-16T19:57:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e1f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e248b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e201f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e1f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42074/0 (socket says 172.21.15.32:42074) 2024-03-16T19:57:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e1f7fe640 1 -- 172.21.15.32:0/3972141763 learned_addr learned my addr 172.21.15.32:0/3972141763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.907+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e20071a50 0x7f6e20148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 -- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 msgr2=0x7f6e201f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e201f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 -- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e20071a50 msgr2=0x7f6e20148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e20071a50 0x7f6e20148cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 -- 172.21.15.32:0/3972141763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e10009420 con 0x7f6e20072520 2024-03-16T19:57:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.911+0000 7f6e1f7fe640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400b410 tx=0x7f6e1400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.915+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14011830 con 0x7f6e20072520 2024-03-16T19:57:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.915+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14011990 con 0x7f6e20072520 2024-03-16T19:57:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.915+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e201f6950 con 0x7f6e20072520 2024-03-16T19:57:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.915+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14011d40 con 0x7f6e20072520 2024-03-16T19:57:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.915+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e201f6f10 con 0x7f6e20072520 2024-03-16T19:57:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.919+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e201f3180 con 0x7f6e20072520 2024-03-16T19:57:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.951+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6e1401a400 con 0x7f6e20072520 2024-03-16T19:57:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.951+0000 7f6e1cff9640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 0x7f6e0c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.951+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 0x7f6e0c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.951+0000 7f6e1ffff640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 0x7f6e0c069330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e100af000 tx=0x7f6e100a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:38.963+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6e14025e30 con 0x7f6e20072520 2024-03-16T19:57:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.251+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6e14051d00 con 0x7f6e20072520 2024-03-16T19:57:39.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.499+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e201f3180 con 0x7f6e20072520 2024-03-16T19:57:39.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e1cff9640 1 -- 172.21.15.32:0/3972141763 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6637 (secure 0 0 0) 0x7f6e201f3180 con 0x7f6e20072520 2024-03-16T19:57:39.486 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:39.486 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:57:25.238491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4246,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4246":{"gid":4246,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":16,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4246,4252]},"id":1}]} 2024-03-16T19:57:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 msgr2=0x7f6e0c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 0x7f6e0c069330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e100af000 tx=0x7f6e100a7000 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 msgr2=0x7f6e20159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400b410 tx=0x7f6e1400b8d0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 shutdown_connections 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e0c066e70 0x7f6e0c069330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e20154300 0x7f6e201f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e20072520 0x7f6e20159200 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 --2- 172.21.15.32:0/3972141763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e20071a50 0x7f6e20148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 >> 172.21.15.32:0/3972141763 conn(0x7f6e200cbfe0 msgr2=0x7f6e200cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 shutdown_connections 2024-03-16T19:57:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.515+0000 7f6e26319640 1 -- 172.21.15.32:0/3972141763 wait complete. 2024-03-16T19:57:39.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:57:39.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 Processor -- start 2024-03-16T19:57:39.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 -- start start 2024-03-16T19:57:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18440c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1844158c80 con 0x7f18440c81e0 2024-03-16T19:57:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184415bfc0 con 0x7f1844151ef0 2024-03-16T19:57:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.639+0000 7f1849c34640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18440b0ad0 con 0x7f184415b9b0 2024-03-16T19:57:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.643+0000 7f18437fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1844151ef0 0x7f18440c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:52484/0 (socket says 172.21.15.32:52484) 2024-03-16T19:57:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.643+0000 7f18437fe640 1 -- 172.21.15.32:0/3608680785 learned_addr learned my addr 172.21.15.32:0/3608680785 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:57:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.643+0000 7f1843fff640 1 --2- 172.21.15.32:0/3608680785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18440c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:39.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.647+0000 7f18427fc640 1 -- 172.21.15.32:0/3608680785 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434378397 0 0) 0x7f1844158c80 con 0x7f18440c81e0 2024-03-16T19:57:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.647+0000 7f18427fc640 1 -- 172.21.15.32:0/3608680785 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f182c003530 con 0x7f18440c81e0 2024-03-16T19:57:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.651+0000 7f1843fff640 1 -- 172.21.15.32:0/3608680785 >> v1:172.21.15.59:6789/0 conn(0x7f1844151ef0 legacy=0x7f18440c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.651+0000 7f1843fff640 1 -- 172.21.15.32:0/3608680785 >> v1:172.21.15.32:6789/0 conn(0x7f18440c81e0 legacy=0x7f184415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.651+0000 7f1843fff640 1 -- 172.21.15.32:0/3608680785 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844157ec0 con 0x7f184415b9b0 2024-03-16T19:57:39.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.655+0000 7f1843fff640 1 --2- 172.21.15.32:0/3608680785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18440c8630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f183c009dc0 tx=0x7f183c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23203fc9a7f73413 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f18427fc640 1 -- 172.21.15.32:0/3608680785 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c0af600 con 0x7f184415b9b0 2024-03-16T19:57:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f18427fc640 1 -- 172.21.15.32:0/3608680785 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c0ab050 con 0x7f184415b9b0 2024-03-16T19:57:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f18427fc640 1 -- 172.21.15.32:0/3608680785 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c0af980 con 0x7f184415b9b0 2024-03-16T19:57:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- 172.21.15.32:0/3608680785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 msgr2=0x7f18440c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 --2- 172.21.15.32:0/3608680785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18440c8630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f183c009dc0 tx=0x7f183c01f170 comp rx=0 tx=0).stop 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- 172.21.15.32:0/3608680785 shutdown_connections 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 --2- 172.21.15.32:0/3608680785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18440c8630 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- 172.21.15.32:0/3608680785 >> 172.21.15.32:0/3608680785 conn(0x7f1844070f90 msgr2=0x7f18440713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- 172.21.15.32:0/3608680785 shutdown_connections 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- 172.21.15.32:0/3608680785 wait complete. 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 Processor -- start 2024-03-16T19:57:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.667+0000 7f1849c34640 1 -- start start 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1844151ef0 0x7f18440c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18441f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18441f2d50 con 0x7f18440c81e0 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18440b2d20 con 0x7f1844151ef0 2024-03-16T19:57:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f1849c34640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18440b13a0 con 0x7f184415b9b0 2024-03-16T19:57:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f18437fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f18437fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54458/0 (socket says 172.21.15.32:54458) 2024-03-16T19:57:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.671+0000 7f18437fe640 1 -- 172.21.15.32:0/1431058592 learned_addr learned my addr 172.21.15.32:0/1431058592 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.675+0000 7f18437fe640 1 -- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 msgr2=0x7f18441f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.675+0000 7f18437fe640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18441f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.675+0000 7f18437fe640 1 -- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1844151ef0 msgr2=0x7f18440c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.675+0000 7f18437fe640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1844151ef0 0x7f18440c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.675+0000 7f18437fe640 1 -- 172.21.15.32:0/1431058592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c009a70 con 0x7f18440c81e0 2024-03-16T19:57:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.679+0000 7f18437fe640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f183400c420 tx=0x7f183400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:39.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.807+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834005cf0 con 0x7f18440c81e0 2024-03-16T19:57:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.807+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1834005e50 con 0x7f18440c81e0 2024-03-16T19:57:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.807+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18440b1b70 con 0x7f18440c81e0 2024-03-16T19:57:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.807+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18340072f0 con 0x7f18440c81e0 2024-03-16T19:57:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.807+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184415b1b0 con 0x7f18440c81e0 2024-03-16T19:57:39.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:39.811+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18441f2e90 con 0x7f18440c81e0 2024-03-16T19:57:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.103+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1834007450 con 0x7f18440c81e0 2024-03-16T19:57:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.103+0000 7f182bfff640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 0x7f1830069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.103+0000 7f1842ffd640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 0x7f1830069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.103+0000 7f1842ffd640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 0x7f1830069380 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1838007fd0 tx=0x7f1838007960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:40.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.127+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f18340556d0 con 0x7f18440c81e0 2024-03-16T19:57:40.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.127+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1834051270 con 0x7f18440c81e0 2024-03-16T19:57:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.435+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1844068000 con 0x7f18440c81e0 2024-03-16T19:57:40.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.523+0000 7f182bfff640 1 -- 172.21.15.32:0/1431058592 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18842 (secure 0 0 0) 0x7f1834017020 con 0x7f18440c81e0 2024-03-16T19:57:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:40.495 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:57:39.778142+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:57:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 msgr2=0x7f1830069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 0x7f1830069380 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1838007fd0 tx=0x7f1838007960 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 msgr2=0x7f18440b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f183400c420 tx=0x7f183400c8e0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 shutdown_connections 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1830066ec0 0x7f1830069380 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f184415b9b0 0x7f18441f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1844151ef0 0x7f18440c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 --2- 172.21.15.32:0/1431058592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18440c81e0 0x7f18440b27e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 >> 172.21.15.32:0/1431058592 conn(0x7f1844070f90 msgr2=0x7f18440c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 shutdown_connections 2024-03-16T19:57:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.527+0000 7f1849c34640 1 -- 172.21.15.32:0/1431058592 wait complete. 2024-03-16T19:57:40.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:57:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.651+0000 7f015c3f1640 1 Processor -- start 2024-03-16T19:57:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.651+0000 7f015c3f1640 1 -- start start 2024-03-16T19:57:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015c3f1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0154150d70 0x7f0154073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015c3f1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01540725e0 0x7f0154070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015c3f1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154072dd0 con 0x7f01540708a0 2024-03-16T19:57:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015c3f1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015415d790 con 0x7f01540725e0 2024-03-16T19:57:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015c3f1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015415da50 con 0x7f0154150d70 2024-03-16T19:57:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015998c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01540725e0 0x7f0154070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015a18d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0154150d70 0x7f0154073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015998c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01540725e0 0x7f0154070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:35188/0 (socket says 172.21.15.32:35188) 2024-03-16T19:57:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.655+0000 7f015998c640 1 -- 172.21.15.32:0/1130651827 learned_addr learned my addr 172.21.15.32:0/1130651827 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.659+0000 7f015998c640 1 -- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0154150d70 msgr2=0x7f0154073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.659+0000 7f015998c640 1 --2- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0154150d70 0x7f0154073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.659+0000 7f015998c640 1 -- 172.21.15.32:0/1130651827 >> v1:172.21.15.59:6789/0 conn(0x7f01540708a0 legacy=0x7f015415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.659+0000 7f015998c640 1 -- 172.21.15.32:0/1130651827 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015415dd30 con 0x7f01540725e0 2024-03-16T19:57:40.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.667+0000 7f015998c640 1 --2- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01540725e0 0x7f0154070360 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58627b8ff1ddd74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015918b640 1 -- 172.21.15.32:0/1130651827 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440ab650 con 0x7f01540725e0 2024-03-16T19:57:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015918b640 1 -- 172.21.15.32:0/1130651827 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01440ab7b0 con 0x7f01540725e0 2024-03-16T19:57:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015918b640 1 -- 172.21.15.32:0/1130651827 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440abb40 con 0x7f01540725e0 2024-03-16T19:57:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01540725e0 msgr2=0x7f0154070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01540725e0 0x7f0154070360 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c490 comp rx=0 tx=0).stop 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- 172.21.15.32:0/1130651827 shutdown_connections 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01540725e0 0x7f0154070360 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1130651827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0154150d70 0x7f0154073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- 172.21.15.32:0/1130651827 >> 172.21.15.32:0/1130651827 conn(0x7f015406c0d0 msgr2=0x7f015406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- 172.21.15.32:0/1130651827 shutdown_connections 2024-03-16T19:57:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- 172.21.15.32:0/1130651827 wait complete. 2024-03-16T19:57:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 Processor -- start 2024-03-16T19:57:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.679+0000 7f015c3f1640 1 -- start start 2024-03-16T19:57:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01540725e0 0x7f015415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0154150d70 0x7f0154149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015414a780 con 0x7f01540708a0 2024-03-16T19:57:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015415c570 con 0x7f0154150d70 2024-03-16T19:57:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015c3f1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015415d1e0 con 0x7f01540725e0 2024-03-16T19:57:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015a18d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015a18d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54462/0 (socket says 172.21.15.32:54462) 2024-03-16T19:57:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015a18d640 1 -- 172.21.15.32:0/1019377733 learned_addr learned my addr 172.21.15.32:0/1019377733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015a98e640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0154150d70 0x7f0154149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.683+0000 7f015998c640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01540725e0 0x7f015415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.687+0000 7f015a18d640 1 -- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01540725e0 msgr2=0x7f015415cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.687+0000 7f015a18d640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01540725e0 0x7f015415cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.687+0000 7f015a18d640 1 -- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0154150d70 msgr2=0x7f0154149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.687+0000 7f015a18d640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0154150d70 0x7f0154149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.687+0000 7f015a18d640 1 -- 172.21.15.32:0/1019377733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440094d0 con 0x7f01540708a0 2024-03-16T19:57:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.691+0000 7f015a18d640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f015000e870 tx=0x7f015000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150009510 con 0x7f01540708a0 2024-03-16T19:57:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000eec0 con 0x7f01540708a0 2024-03-16T19:57:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015000cb40 con 0x7f01540708a0 2024-03-16T19:57:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015415d560 con 0x7f01540708a0 2024-03-16T19:57:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01541484f0 con 0x7f01540708a0 2024-03-16T19:57:40.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.695+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0154068000 con 0x7f01540708a0 2024-03-16T19:57:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.975+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f015000cca0 con 0x7f01540708a0 2024-03-16T19:57:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.975+0000 7f014affd640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 0x7f0138069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.975+0000 7f015998c640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 0x7f0138069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.975+0000 7f015998c640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 0x7f0138069300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01440af000 tx=0x7f01440b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:40.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.995+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f0150055ee0 con 0x7f01540708a0 2024-03-16T19:57:40.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:40.995+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0150052180 con 0x7f01540708a0 2024-03-16T19:57:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.291+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01540c75b0 con 0x7f01540708a0 2024-03-16T19:57:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.303+0000 7f014affd640 1 -- 172.21.15.32:0/1019377733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6637 (secure 0 0 0) 0x7f0150017090 con 0x7f01540708a0 2024-03-16T19:57:41.277 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:57:41.277 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:57:25.238491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4246,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4246":{"gid":4246,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":16,"addr":"172.21.15.59:6835/1407783290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1407783290},{"type":"v1","addr":"172.21.15.59:6835","nonce":1407783290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4246,4252]},"id":1}]} 2024-03-16T19:57:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-03-16T19:57:41.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 msgr2=0x7f0138069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 0x7f0138069300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01440af000 tx=0x7f01440b6040 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 msgr2=0x7f01541f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f015000e870 tx=0x7f015000ec70 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 shutdown_connections 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0138066e40 0x7f0138069300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0154150d70 0x7f0154149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01540725e0 0x7f015415cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 --2- 172.21.15.32:0/1019377733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01540708a0 0x7f01541f3210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 >> 172.21.15.32:0/1019377733 conn(0x7f015406c0d0 msgr2=0x7f01540cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 shutdown_connections 2024-03-16T19:57:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:41.307+0000 7f015c3f1640 1 -- 172.21.15.32:0/1019377733 wait complete. 2024-03-16T19:57:41.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:57:41.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:57:41.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:57:25.238491+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4246': {'gid': 4246, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.59:6835/1407783290', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1407783290}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1407783290}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4246, 4252]} 2024-03-16T19:57:41.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-16T19:57:41.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-16T19:57:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 Processor -- start 2024-03-16T19:57:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 -- start start 2024-03-16T19:57:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a18150e20 0x7f9a18073060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c7590 0x7f9a180c2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a180c2eb0 0x7f9a181572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a18156bb0 con 0x7f9a18150e20 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a18072e60 con 0x7f9a180c2eb0 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1f297640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a180735a0 con 0x7f9a180c7590 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.043+0000 7f9a1d033640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a18150e20 0x7f9a18073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a18150e20 0x7f9a18073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54464/0 (socket says 172.21.15.32:54464) 2024-03-16T19:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 -- 172.21.15.32:0/25636819 learned_addr learned my addr 172.21.15.32:0/25636819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d834640 1 --2- 172.21.15.32:0/25636819 >> v2:172.21.15.59:3300/0 conn(0x7f9a180c2eb0 0x7f9a181572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1c832640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c7590 0x7f9a180c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 -- 172.21.15.32:0/25636819 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c7590 msgr2=0x7f9a180c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c7590 0x7f9a180c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 -- 172.21.15.32:0/25636819 >> v2:172.21.15.59:3300/0 conn(0x7f9a180c2eb0 msgr2=0x7f9a181572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 --2- 172.21.15.32:0/25636819 >> v2:172.21.15.59:3300/0 conn(0x7f9a180c2eb0 0x7f9a181572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.047+0000 7f9a1d033640 1 -- 172.21.15.32:0/25636819 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a18156dc0 con 0x7f9a18150e20 2024-03-16T19:57:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.051+0000 7f9a1d033640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a18150e20 0x7f9a18073060 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a1815b240 tx=0x7f9a1401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6489c2518e77dfc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a0ffff640 1 -- 172.21.15.32:0/25636819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140ac620 con 0x7f9a18150e20 2024-03-16T19:57:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a0ffff640 1 -- 172.21.15.32:0/25636819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a140ac780 con 0x7f9a18150e20 2024-03-16T19:57:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a0ffff640 1 -- 172.21.15.32:0/25636819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140acb10 con 0x7f9a18150e20 2024-03-16T19:57:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- 172.21.15.32:0/25636819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a18150e20 msgr2=0x7f9a18073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a18150e20 0x7f9a18073060 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a1815b240 tx=0x7f9a1401c4f0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- 172.21.15.32:0/25636819 shutdown_connections 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 --2- 172.21.15.32:0/25636819 >> v2:172.21.15.59:3300/0 conn(0x7f9a180c2eb0 0x7f9a181572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c7590 0x7f9a180c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 --2- 172.21.15.32:0/25636819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a18150e20 0x7f9a18073060 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- 172.21.15.32:0/25636819 >> 172.21.15.32:0/25636819 conn(0x7f9a1806afc0 msgr2=0x7f9a1806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- 172.21.15.32:0/25636819 shutdown_connections 2024-03-16T19:57:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- 172.21.15.32:0/25636819 wait complete. 2024-03-16T19:57:42.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 Processor -- start 2024-03-16T19:57:42.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.059+0000 7f9a1f297640 1 -- start start 2024-03-16T19:57:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c2eb0 0x7f9a181f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a180c7590 0x7f9a181410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a18150e20 0x7f9a181407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a181f6af0 con 0x7f9a180c7590 2024-03-16T19:57:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1814bef0 con 0x7f9a18150e20 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1f297640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1814c6e0 con 0x7f9a180c2eb0 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1c832640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a180c7590 0x7f9a181410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1c832640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a180c7590 0x7f9a181410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54468/0 (socket says 172.21.15.32:54468) 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1c832640 1 -- 172.21.15.32:0/2796187436 learned_addr learned my addr 172.21.15.32:0/2796187436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1d834640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a18150e20 0x7f9a181407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.063+0000 7f9a1d033640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c2eb0 0x7f9a181f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.067+0000 7f9a1c832640 1 -- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c2eb0 msgr2=0x7f9a181f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.067+0000 7f9a1c832640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a180c2eb0 0x7f9a181f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.067+0000 7f9a1c832640 1 -- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a18150e20 msgr2=0x7f9a181407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.067+0000 7f9a1c832640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a18150e20 0x7f9a181407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.067+0000 7f9a1c832640 1 -- 172.21.15.32:0/2796187436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140094d0 con 0x7f9a180c7590 2024-03-16T19:57:42.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.071+0000 7f9a1c832640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a180c7590 0x7f9a181410d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800e870 tx=0x7f9a0800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.075+0000 7f9a0dffb640 1 -- 172.21.15.32:0/2796187436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a08009510 con 0x7f9a180c7590 2024-03-16T19:57:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.075+0000 7f9a0dffb640 1 -- 172.21.15.32:0/2796187436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0800eec0 con 0x7f9a180c7590 2024-03-16T19:57:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.075+0000 7f9a0dffb640 1 -- 172.21.15.32:0/2796187436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0800cb40 con 0x7f9a180c7590 2024-03-16T19:57:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.075+0000 7f9a1f297640 1 -- 172.21.15.32:0/2796187436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1814cc80 con 0x7f9a180c7590 2024-03-16T19:57:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.079+0000 7f9a1f297640 1 -- 172.21.15.32:0/2796187436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a181f64a0 con 0x7f9a180c7590 2024-03-16T19:57:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.079+0000 7f9a22409000 1 Processor -- start 2024-03-16T19:57:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.079+0000 7f9a22409000 1 -- start start 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85de7890 0x560d85e22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e22bc0 0x560d85e22fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85ca9d00 con 0x560d85de7890 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85c3b430 con 0x560d85e20f70 2024-03-16T19:57:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a22409000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d85c38960 con 0x560d85e22bc0 2024-03-16T19:57:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a1d834640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a1c832640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e22bc0 0x560d85e22fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a1d834640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:35228/0 (socket says 172.21.15.32:35228) 2024-03-16T19:57:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.083+0000 7f9a1d834640 1 -- 172.21.15.32:0/1552056385 learned_addr learned my addr 172.21.15.32:0/1552056385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.087+0000 7f9a1d834640 1 -- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e22bc0 msgr2=0x560d85e22fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.087+0000 7f9a1d834640 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e22bc0 0x560d85e22fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.087+0000 7f9a1d834640 1 -- 172.21.15.32:0/1552056385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85de7890 msgr2=0x560d85e22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.087+0000 7f9a1d834640 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85de7890 0x560d85e22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.087+0000 7f9a1d834640 1 -- 172.21.15.32:0/1552056385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d8592a890 con 0x560d85e20f70 2024-03-16T19:57:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.091+0000 7f9a1d834640 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9a10011230 tx=0x7f9a1000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b739560d4db97ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f99faffd640 1 -- 172.21.15.32:0/1552056385 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10010040 con 0x560d85e20f70 2024-03-16T19:57:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f99faffd640 1 -- 172.21.15.32:0/1552056385 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10016860 con 0x560d85e20f70 2024-03-16T19:57:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f99faffd640 1 -- 172.21.15.32:0/1552056385 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10016bd0 con 0x560d85e20f70 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 -- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 msgr2=0x560d85de7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9a10011230 tx=0x7f9a1000b040 comp rx=0 tx=0).stop 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 -- 172.21.15.32:0/1552056385 shutdown_connections 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e22bc0 0x560d85e22fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85de7890 0x560d85e22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 --2- 172.21.15.32:0/1552056385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85e20f70 0x560d85de7350 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 -- 172.21.15.32:0/1552056385 >> 172.21.15.32:0/1552056385 conn(0x560d85de7ff0 msgr2=0x560d85e20440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:57:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 -- 172.21.15.32:0/1552056385 shutdown_connections 2024-03-16T19:57:42.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.099+0000 7f9a22409000 1 -- 172.21.15.32:0/1552056385 wait complete. 2024-03-16T19:57:42.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 -- start start 2024-03-16T19:57:42.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 Processor -- start 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85de7890 0x560d85e2dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e20f70 0x560d85e2e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85e22bc0 0x560d85e1cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85c34140 con 0x560d85e22bc0 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85bce630 con 0x560d85de7890 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d85bce2d0 con 0x560d85e20f70 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a22409000 10 client.0 _ll_register_callbacks cb 0x560d85ce41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a0dffb640 1 -- 172.21.15.32:0/2796187436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9a080059a0 con 0x7f9a180c7590 2024-03-16T19:57:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a1d033640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e20f70 0x560d85e2e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a0dffb640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f99fc066e00 0x7f99fc0692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a1d033640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e20f70 0x560d85e2e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:54008/0 (socket says 172.21.15.32:54008) 2024-03-16T19:57:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a1d033640 1 -- 172.21.15.32:0/3309540721 learned_addr learned my addr 172.21.15.32:0/3309540721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:57:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.103+0000 7f9a1d834640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f99fc066e00 0x7f99fc0692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.107+0000 7f9a1d834640 1 --2- 172.21.15.32:0/2796187436 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f99fc066e00 0x7f99fc0692c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009d00 tx=0x7f9a10014000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.107+0000 7f9a1d834640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85de7890 0x560d85e2dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:57:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.107+0000 7f9a0dffb640 1 -- 172.21.15.32:0/2796187436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9a08055980 con 0x7f9a180c7590 2024-03-16T19:57:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.111+0000 7f9a1d033640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85de7890 msgr2=0x560d85e2dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.111+0000 7f9a1d033640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560d85de7890 0x560d85e2dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.111+0000 7f9a1d033640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85e22bc0 msgr2=0x560d85e1cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:57:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.111+0000 7f9a1d033640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d85e22bc0 0x560d85e1cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:57:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.111+0000 7f9a1d033640 1 -- 172.21.15.32:0/3309540721 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a10009d30 con 0x560d85e20f70 2024-03-16T19:57:42.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.115+0000 7f9a1d033640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x560d85e20f70 0x560d85e2e2b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a140af040 tx=0x7f9a1401ad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df9f48a12639190e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:57:42.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.115+0000 7f99f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3301/0 2024-03-16T19:57:42.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.123+0000 7f99f3fff640 1 -- 172.21.15.32:0/3309540721 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140acbb0 con 0x560d85e20f70 2024-03-16T19:57:42.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.123+0000 7f99f3fff640 1 -- 172.21.15.32:0/3309540721 <== mon.2 v2:172.21.15.59:3301/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9a140acd10 con 0x560d85e20f70 2024-03-16T19:57:42.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.123+0000 7f9a22409000 1 -- 172.21.15.32:0/3309540721 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d85be7850 con 0x560d85e20f70 2024-03-16T19:57:42.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.127+0000 7f99f3fff640 1 -- 172.21.15.32:0/3309540721 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a140b6410 con 0x560d85e20f70 2024-03-16T19:57:42.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.127+0000 7f9a22409000 10 client.4550 fetch_fsmap learned FSMap version 18 2024-03-16T19:57:42.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.127+0000 7f9a22409000 1 -- 172.21.15.32:0/3309540721 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x560d85bf3e70 con 0x560d85e20f70 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f99f3fff640 1 -- 172.21.15.32:0/3309540721 <== mon.2 v2:172.21.15.59:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560d85be7850 con 0x560d85e20f70 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f99f3fff640 1 -- 172.21.15.32:0/3309540721 <== mon.2 v2:172.21.15.59:3301/0 5 ==== fsmap(e 18) v1 ==== 4552+0+0 (secure 0 0 0) 0x7f9a140b6570 con 0x560d85e20f70 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 10 client.4550 fetch_fsmap finished waiting for FSMap version 18 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 10 client.4550 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 20 client.4550 populate_metadata read hostname 'smithi032' 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 4 client.4550 mds_command: new command op to 4242 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T19:57:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:42.155+0000 7f9a22409000 1 -- 172.21.15.32:0/3309540721 --> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d85c30f20 con 0x560d85e2c140 2024-03-16T19:57:52.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:52.163+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:57:52.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:57:52.163+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:58:02.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:02.375+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:58:02.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:02.375+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T19:58:11.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:58:11.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:58:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:12.787+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:58:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:12.787+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T19:58:21.526 INFO:tasks.ceph.mds.f.smithi059.stderr:2024-03-16T19:58:21.549+0000 13df2640 -1 mds.2.openfiles _load_finish got (2) No such file or directory 2024-03-16T19:58:23.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:23.603+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:58:23.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:23.603+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T19:58:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:35.210+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:58:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:35.210+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T19:58:41.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:58:41.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:58:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:48.418+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:58:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:58:48.418+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-16T19:59:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:04.834+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:59:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:04.834+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-16T19:59:11.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:59:11.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:59:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:27.642+0000 7f9a1c832640 1 -- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 msgr2=0x560d85e2c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.32:6836/3949943837, fault. 2024-03-16T19:59:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:27.642+0000 7f9a1c832640 1 --2- 172.21.15.32:0/3309540721 >> [v2:172.21.15.32:6836/3949943837,v1:172.21.15.32:6837/3949943837] conn(0x560d85e2c140 0x560d85e2c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-16T19:59:41.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:59:41.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:59:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-16T19:59:42.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.057+0000 7f9cdd408640 1 Processor -- start 2024-03-16T19:59:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.057+0000 7f9cdd408640 1 -- start start 2024-03-16T19:59:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cdd408640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9cd8158980 0x7f9cd8163740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cdd408640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd80c68e0 con 0x7f9cd80b1eb0 2024-03-16T19:59:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cdd408640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd8067340 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cdd408640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd806c1f0 con 0x7f9cd8158980 2024-03-16T19:59:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cd6ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9cd80b1eb0 0x7f9cd8072a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36216/0 (socket says 172.21.15.32:36216) 2024-03-16T19:59:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cd6ffd640 1 -- 172.21.15.32:0/1870424049 learned_addr learned my addr 172.21.15.32:0/1870424049 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:59:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.061+0000 7f9cd77fe640 1 --2- 172.21.15.32:0/1870424049 >> v2:172.21.15.59:3300/0 conn(0x7f9cd8158980 0x7f9cd8163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.065+0000 7f9cd5ffb640 1 -- 172.21.15.32:0/1870424049 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519890109 0 0) 0x7f9cd80c68e0 con 0x7f9cd80b1eb0 2024-03-16T19:59:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.065+0000 7f9cd5ffb640 1 -- 172.21.15.32:0/1870424049 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc0003530 con 0x7f9cd80b1eb0 2024-03-16T19:59:43.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.069+0000 7f9cd77fe640 1 -- 172.21.15.32:0/1870424049 >> v1:172.21.15.59:6789/0 conn(0x7f9cd80c7ce0 legacy=0x7f9cd8158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:43.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.069+0000 7f9cd77fe640 1 -- 172.21.15.32:0/1870424049 >> v1:172.21.15.32:6789/0 conn(0x7f9cd80b1eb0 legacy=0x7f9cd8072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.069+0000 7f9cd77fe640 1 -- 172.21.15.32:0/1870424049 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd806c4d0 con 0x7f9cd8158980 2024-03-16T19:59:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.073+0000 7f9cd77fe640 1 --2- 172.21.15.32:0/1870424049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd8158980 0x7f9cd8163740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009d90 tx=0x7f9cd001f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce89fef95160c4f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cd5ffb640 1 -- 172.21.15.32:0/1870424049 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00af600 con 0x7f9cd8158980 2024-03-16T19:59:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cd5ffb640 1 -- 172.21.15.32:0/1870424049 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd00ab050 con 0x7f9cd8158980 2024-03-16T19:59:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cd5ffb640 1 -- 172.21.15.32:0/1870424049 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00af980 con 0x7f9cd8158980 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- 172.21.15.32:0/1870424049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd8158980 msgr2=0x7f9cd8163740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1870424049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd8158980 0x7f9cd8163740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009d90 tx=0x7f9cd001f120 comp rx=0 tx=0).stop 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- 172.21.15.32:0/1870424049 shutdown_connections 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1870424049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd8158980 0x7f9cd8163740 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- 172.21.15.32:0/1870424049 >> 172.21.15.32:0/1870424049 conn(0x7f9cd806afc0 msgr2=0x7f9cd806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- 172.21.15.32:0/1870424049 shutdown_connections 2024-03-16T19:59:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- 172.21.15.32:0/1870424049 wait complete. 2024-03-16T19:59:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 Processor -- start 2024-03-16T19:59:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.085+0000 7f9cdd408640 1 -- start start 2024-03-16T19:59:43.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd80b1eb0 0x7f9cd8156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:43.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cd8158980 0x7f9cd80c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd81f3b00 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd81594e0 con 0x7f9cd80b1eb0 2024-03-16T19:59:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cdd408640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd814dda0 con 0x7f9cd8158980 2024-03-16T19:59:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cd67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cd67fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46748/0 (socket says 172.21.15.32:46748) 2024-03-16T19:59:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cd67fc640 1 -- 172.21.15.32:0/1829449459 learned_addr learned my addr 172.21.15.32:0/1829449459 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cd6ffd640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd80b1eb0 0x7f9cd8156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.089+0000 7f9cd77fe640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cd8158980 0x7f9cd80c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 -- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cd8158980 msgr2=0x7f9cd80c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cd8158980 0x7f9cd80c1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 -- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd80b1eb0 msgr2=0x7f9cd8156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd80b1eb0 0x7f9cd8156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 -- 172.21.15.32:0/1829449459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd0009a70 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.093+0000 7f9cd67fc640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c410 tx=0x7f9ccc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc003600 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc003780 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd81f2ec0 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc003b30 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd81f3160 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.101+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd81f3c40 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.377+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9ccc003c90 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.377+0000 7f9cbf7fe640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 0x7f9cc4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.377+0000 7f9cd6ffd640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 0x7f9cc4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.377+0000 7f9cd6ffd640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 0x7f9cc4069300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8007fd0 tx=0x7f9cc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:43.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.393+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9ccc0010f0 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.393+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9ccc0514c0 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.613+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cd81f3c40 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.637+0000 7f9cbf7fe640 1 -- 172.21.15.32:0/1829449459 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f9cd81f3c40 con 0x7f9cd80c7ce0 2024-03-16T19:59:43.612 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:43.612 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 msgr2=0x7f9cc4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 0x7f9cc4069300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8007fd0 tx=0x7f9cc8009040 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 msgr2=0x7f9cd80cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c410 tx=0x7f9ccc00c8d0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 shutdown_connections 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9cc4066e40 0x7f9cc4069300 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cd8158980 0x7f9cd80c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd80c7ce0 0x7f9cd80cab80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 --2- 172.21.15.32:0/1829449459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd80b1eb0 0x7f9cd8156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 >> 172.21.15.32:0/1829449459 conn(0x7f9cd806afc0 msgr2=0x7f9cd8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 shutdown_connections 2024-03-16T19:59:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:43.641+0000 7f9cdd408640 1 -- 172.21.15.32:0/1829449459 wait complete. 2024-03-16T19:59:44.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:44.234 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:45.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 Processor -- start 2024-03-16T19:59:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 -- start start 2024-03-16T19:59:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f4c0c6c40 0x7f5f4c06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0c7020 0x7f5f4c158380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c157ef0 con 0x7f5f4c1588c0 2024-03-16T19:59:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c157c50 con 0x7f5f4c0c7020 2024-03-16T19:59:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.369+0000 7f5f52fdc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c06eed0 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.373+0000 7f5f50d78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f4c0c6c40 0x7f5f4c06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.373+0000 7f5f50d78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f4c0c6c40 0x7f5f4c06e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:34578/0 (socket says 172.21.15.32:34578) 2024-03-16T19:59:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.373+0000 7f5f50d78640 1 -- 172.21.15.32:0/991809830 learned_addr learned my addr 172.21.15.32:0/991809830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:45.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.373+0000 7f5f4bfff640 1 --2- 172.21.15.32:0/991809830 >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0c7020 0x7f5f4c158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.377+0000 7f5f50d78640 1 -- 172.21.15.32:0/991809830 >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0c7020 msgr2=0x7f5f4c158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:45.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.377+0000 7f5f50d78640 1 --2- 172.21.15.32:0/991809830 >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0c7020 0x7f5f4c158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.377+0000 7f5f50d78640 1 -- 172.21.15.32:0/991809830 >> v1:172.21.15.32:6789/0 conn(0x7f5f4c1588c0 legacy=0x7f5f4c151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.377+0000 7f5f50d78640 1 -- 172.21.15.32:0/991809830 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c072b50 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.381+0000 7f5f50d78640 1 --2- 172.21.15.32:0/991809830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c06e870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0cce50 tx=0x7f5f4001c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a84249a48ef079b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f4b7fe640 1 -- 172.21.15.32:0/991809830 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f400aa650 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f4b7fe640 1 -- 172.21.15.32:0/991809830 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f400aa7b0 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f4b7fe640 1 -- 172.21.15.32:0/991809830 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f400aab40 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- 172.21.15.32:0/991809830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c6c40 msgr2=0x7f5f4c06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/991809830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c06e870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0cce50 tx=0x7f5f4001c4f0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- 172.21.15.32:0/991809830 shutdown_connections 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/991809830 >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0c7020 0x7f5f4c158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/991809830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c06e870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- 172.21.15.32:0/991809830 >> 172.21.15.32:0/991809830 conn(0x7f5f4c0cbfe0 msgr2=0x7f5f4c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- 172.21.15.32:0/991809830 shutdown_connections 2024-03-16T19:59:45.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- 172.21.15.32:0/991809830 wait complete. 2024-03-16T19:59:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 Processor -- start 2024-03-16T19:59:45.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.393+0000 7f5f52fdc640 1 -- start start 2024-03-16T19:59:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c7020 0x7f5f4c14d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5f4c1588c0 0x7f5f4c14dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c14d360 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c143ae0 con 0x7f5f4c0c7020 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f52fdc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c14cd30 con 0x7f5f4c1588c0 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f50d78640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f50d78640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46768/0 (socket says 172.21.15.32:46768) 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f50d78640 1 -- 172.21.15.32:0/3669173109 learned_addr learned my addr 172.21.15.32:0/3669173109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.397+0000 7f5f51579640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5f4c1588c0 0x7f5f4c14dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f4bfff640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c7020 0x7f5f4c14d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 -- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5f4c1588c0 msgr2=0x7f5f4c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5f4c1588c0 0x7f5f4c14dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 -- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c7020 msgr2=0x7f5f4c14d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c7020 0x7f5f4c14d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 -- 172.21.15.32:0/3669173109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400094d0 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.401+0000 7f5f50d78640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5f4001c9d0 tx=0x7f5f4001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.405+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f400b3800 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.405+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f400b3960 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.405+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c155510 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.409+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f400b3d10 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.409+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c0c4460 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.409+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c14d5d0 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.437+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5f400c7420 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.437+0000 7f5f497fa640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 0x7f5f30069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.437+0000 7f5f4bfff640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 0x7f5f30069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.441+0000 7f5f4bfff640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 0x7f5f30069330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c144650 tx=0x7f5f38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:45.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.729+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f5f400f00a0 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.729+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5f400ec340 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.953+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c068000 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.977+0000 7f5f497fa640 1 -- 172.21.15.32:0/3669173109 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f5f400efa00 con 0x7f5f4c0c6c40 2024-03-16T19:59:45.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:45.952 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:45.952 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 msgr2=0x7f5f30069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 0x7f5f30069330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c144650 tx=0x7f5f38008040 comp rx=0 tx=0).stop 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 msgr2=0x7f5f4c1435a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5f4001c9d0 tx=0x7f5f4001ad30 comp rx=0 tx=0).stop 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 shutdown_connections 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5f30066e70 0x7f5f30069330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5f4c1588c0 0x7f5f4c14dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f4c0c7020 0x7f5f4c14d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 --2- 172.21.15.32:0/3669173109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0c6c40 0x7f5f4c1435a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:45.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 >> 172.21.15.32:0/3669173109 conn(0x7f5f4c0cbfe0 msgr2=0x7f5f4c156970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:45.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 shutdown_connections 2024-03-16T19:59:45.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:45.981+0000 7f5f52fdc640 1 -- 172.21.15.32:0/3669173109 wait complete. 2024-03-16T19:59:46.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:46.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:47.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 Processor -- start 2024-03-16T19:59:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 -- start start 2024-03-16T19:59:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5c8151000 0x7fb5c80c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c80c36b0 0x7fb5c815a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5c80c8980 0x7fb5c80c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c8072f10 con 0x7fb5c8151000 2024-03-16T19:59:47.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c80c5000 con 0x7fb5c80c8980 2024-03-16T19:59:47.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5ceccb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c8ec0 con 0x7fb5c80c36b0 2024-03-16T19:59:47.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5cca67640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5c8151000 0x7fb5c80c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5cca67640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5c8151000 0x7fb5c80c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46780/0 (socket says 172.21.15.32:46780) 2024-03-16T19:59:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.713+0000 7fb5cca67640 1 -- 172.21.15.32:0/581445666 learned_addr learned my addr 172.21.15.32:0/581445666 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cd268640 1 --2- 172.21.15.32:0/581445666 >> v2:172.21.15.59:3300/0 conn(0x7fb5c80c8980 0x7fb5c80c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5c7fff640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c80c36b0 0x7fb5c815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cca67640 1 -- 172.21.15.32:0/581445666 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c80c36b0 msgr2=0x7fb5c815a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cca67640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c80c36b0 0x7fb5c815a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cca67640 1 -- 172.21.15.32:0/581445666 >> v2:172.21.15.59:3300/0 conn(0x7fb5c80c8980 msgr2=0x7fb5c80c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cca67640 1 --2- 172.21.15.32:0/581445666 >> v2:172.21.15.59:3300/0 conn(0x7fb5c80c8980 0x7fb5c80c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.717+0000 7fb5cca67640 1 -- 172.21.15.32:0/581445666 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c8072950 con 0x7fb5c8151000 2024-03-16T19:59:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.721+0000 7fb5cca67640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c8151000 0x7fb5c80c8420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb5c806dbd0 tx=0x7fb5bc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6835562dea86266f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.725+0000 7fb5c77fe640 1 -- 172.21.15.32:0/581445666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0aa640 con 0x7fb5c8151000 2024-03-16T19:59:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.725+0000 7fb5c77fe640 1 -- 172.21.15.32:0/581445666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc0aa7a0 con 0x7fb5c8151000 2024-03-16T19:59:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.725+0000 7fb5c77fe640 1 -- 172.21.15.32:0/581445666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0aab30 con 0x7fb5c8151000 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- 172.21.15.32:0/581445666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c8151000 msgr2=0x7fb5c80c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c8151000 0x7fb5c80c8420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb5c806dbd0 tx=0x7fb5bc01c520 comp rx=0 tx=0).stop 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- 172.21.15.32:0/581445666 shutdown_connections 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/581445666 >> v2:172.21.15.59:3300/0 conn(0x7fb5c80c8980 0x7fb5c80c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c80c36b0 0x7fb5c815a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/581445666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c8151000 0x7fb5c80c8420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- 172.21.15.32:0/581445666 >> 172.21.15.32:0/581445666 conn(0x7fb5c806c0d0 msgr2=0x7fb5c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- 172.21.15.32:0/581445666 shutdown_connections 2024-03-16T19:59:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- 172.21.15.32:0/581445666 wait complete. 2024-03-16T19:59:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 Processor -- start 2024-03-16T19:59:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- start start 2024-03-16T19:59:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c80c8980 0x7fb5c81471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c8151000 0x7fb5c81f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80cca70 con 0x7fb5c80c36b0 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c81ee5f0 con 0x7fb5c80c8980 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5ceccb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80cc1d0 con 0x7fb5c8151000 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5cca67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5cca67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46796/0 (socket says 172.21.15.32:46796) 2024-03-16T19:59:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5c7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c80c8980 0x7fb5c81471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5cca67640 1 -- 172.21.15.32:0/565225741 learned_addr learned my addr 172.21.15.32:0/565225741 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.729+0000 7fb5cd268640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c8151000 0x7fb5c81f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.733+0000 7fb5cca67640 1 -- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c8151000 msgr2=0x7fb5c81f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.733+0000 7fb5cca67640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c8151000 0x7fb5c81f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.733+0000 7fb5cca67640 1 -- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c80c8980 msgr2=0x7fb5c81471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.733+0000 7fb5cca67640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c80c8980 0x7fb5c81471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.733+0000 7fb5cca67640 1 -- 172.21.15.32:0/565225741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc0094d0 con 0x7fb5c80c36b0 2024-03-16T19:59:47.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.737+0000 7fb5cca67640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc01abd0 tx=0x7fb5bc01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.741+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0bd5f0 con 0x7fb5c80c36b0 2024-03-16T19:59:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.741+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc0bd770 con 0x7fb5c80c36b0 2024-03-16T19:59:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.741+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0bdb20 con 0x7fb5c80c36b0 2024-03-16T19:59:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.741+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c80cc4b0 con 0x7fb5c80c36b0 2024-03-16T19:59:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.741+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c81ee890 con 0x7fb5c80c36b0 2024-03-16T19:59:47.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.745+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c81ee450 con 0x7fb5c80c36b0 2024-03-16T19:59:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.765+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb5bc0a8050 con 0x7fb5c80c36b0 2024-03-16T19:59:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.765+0000 7fb5c57fa640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 0x7fb5ac069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.765+0000 7fb5c7fff640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 0x7fb5ac069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:47.765+0000 7fb5c7fff640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 0x7fb5ac069380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80c79d0 tx=0x7fb5b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.037+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb5bc0effc0 con 0x7fb5c80c36b0 2024-03-16T19:59:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.037+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb5bc0ec260 con 0x7fb5c80c36b0 2024-03-16T19:59:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.237+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5c81ee450 con 0x7fb5c80c36b0 2024-03-16T19:59:48.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.257+0000 7fb5c57fa640 1 -- 172.21.15.32:0/565225741 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fb5c81ee450 con 0x7fb5c80c36b0 2024-03-16T19:59:48.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:48.233 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 msgr2=0x7fb5ac069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 0x7fb5ac069380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80c79d0 tx=0x7fb5b4009190 comp rx=0 tx=0).stop 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 msgr2=0x7fb5c81eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc01abd0 tx=0x7fb5bc01ad60 comp rx=0 tx=0).stop 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 shutdown_connections 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb5ac066ec0 0x7fb5ac069380 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5c8151000 0x7fb5c81f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c80c8980 0x7fb5c81471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 --2- 172.21.15.32:0/565225741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5c80c36b0 0x7fb5c81eedb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 >> 172.21.15.32:0/565225741 conn(0x7fb5c806c0d0 msgr2=0x7fb5c80c5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 shutdown_connections 2024-03-16T19:59:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:48.261+0000 7fb5ceccb640 1 -- 172.21.15.32:0/565225741 wait complete. 2024-03-16T19:59:48.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:48.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:49.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:49.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 Processor -- start 2024-03-16T19:59:49.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 -- start start 2024-03-16T19:59:49.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb230151000 0x7fb2300c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2300c36b0 0x7fb23015a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2300c8980 0x7fb2300c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb230072f10 con 0x7fb2300c8980 2024-03-16T19:59:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2300c5000 con 0x7fb230151000 2024-03-16T19:59:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb234f0c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300c8ec0 con 0x7fb2300c36b0 2024-03-16T19:59:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb22ed9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2300c8980 0x7fb2300c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb22ed9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2300c8980 0x7fb2300c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58416/0 (socket says 172.21.15.32:58416) 2024-03-16T19:59:49.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb22ed9d640 1 -- 172.21.15.32:0/2842666369 learned_addr learned my addr 172.21.15.32:0/2842666369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:49.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb22dd9b640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2300c36b0 0x7fb23015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:49.997+0000 7fb22e59c640 1 --2- 172.21.15.32:0/2842666369 >> v2:172.21.15.59:3300/0 conn(0x7fb230151000 0x7fb2300c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.001+0000 7fb22ed9d640 1 -- 172.21.15.32:0/2842666369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2300c36b0 msgr2=0x7fb23015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.001+0000 7fb22ed9d640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2300c36b0 0x7fb23015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.001+0000 7fb22ed9d640 1 -- 172.21.15.32:0/2842666369 >> v2:172.21.15.59:3300/0 conn(0x7fb230151000 msgr2=0x7fb2300c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.001+0000 7fb22ed9d640 1 --2- 172.21.15.32:0/2842666369 >> v2:172.21.15.59:3300/0 conn(0x7fb230151000 0x7fb2300c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.001+0000 7fb22ed9d640 1 -- 172.21.15.32:0/2842666369 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230072950 con 0x7fb2300c8980 2024-03-16T19:59:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.005+0000 7fb22ed9d640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c8980 0x7fb2300c9160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb2300725d0 tx=0x7fb22801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452d4c372d197a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:49.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.009+0000 7fb22d59a640 1 -- 172.21.15.32:0/2842666369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2280a5870 con 0x7fb2300c8980 2024-03-16T19:59:49.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.009+0000 7fb22d59a640 1 -- 172.21.15.32:0/2842666369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2280a59d0 con 0x7fb2300c8980 2024-03-16T19:59:49.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.009+0000 7fb22d59a640 1 -- 172.21.15.32:0/2842666369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2280a5d60 con 0x7fb2300c8980 2024-03-16T19:59:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- 172.21.15.32:0/2842666369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c8980 msgr2=0x7fb2300c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c8980 0x7fb2300c9160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb2300725d0 tx=0x7fb22801ca10 comp rx=0 tx=0).stop 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- 172.21.15.32:0/2842666369 shutdown_connections 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c8980 0x7fb2300c9160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 --2- 172.21.15.32:0/2842666369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb2300c36b0 0x7fb23015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 --2- 172.21.15.32:0/2842666369 >> v2:172.21.15.59:3300/0 conn(0x7fb230151000 0x7fb2300c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- 172.21.15.32:0/2842666369 >> 172.21.15.32:0/2842666369 conn(0x7fb23006c0d0 msgr2=0x7fb23006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- 172.21.15.32:0/2842666369 shutdown_connections 2024-03-16T19:59:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- 172.21.15.32:0/2842666369 wait complete. 2024-03-16T19:59:49.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 Processor -- start 2024-03-16T19:59:49.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 -- start start 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.013+0000 7fb234f0c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb234f0c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2300c8980 0x7fb2301f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb234f0c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb230151000 0x7fb23006b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb234f0c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300cc790 con 0x7fb2300c36b0 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb234f0c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300cd570 con 0x7fb2300c8980 2024-03-16T19:59:49.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb234f0c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300cbdc0 con 0x7fb230151000 2024-03-16T19:59:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb22e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb22dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2300c8980 0x7fb2301f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb22e59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58422/0 (socket says 172.21.15.32:58422) 2024-03-16T19:59:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb22e59c640 1 -- 172.21.15.32:0/3380066994 learned_addr learned my addr 172.21.15.32:0/3380066994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.017+0000 7fb22ed9d640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb230151000 0x7fb23006b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:49.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 -- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb230151000 msgr2=0x7fb23006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:49.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb230151000 0x7fb23006b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 -- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2300c8980 msgr2=0x7fb2301f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:49.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2300c8980 0x7fb2301f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:49.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 -- 172.21.15.32:0/3380066994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228009b30 con 0x7fb2300c36b0 2024-03-16T19:59:49.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.021+0000 7fb22e59c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb22000e8a0 tx=0x7fb22000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.025+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220009510 con 0x7fb2300c36b0 2024-03-16T19:59:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.029+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22000c8e0 con 0x7fb2300c36b0 2024-03-16T19:59:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.029+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2301f2570 con 0x7fb2300c36b0 2024-03-16T19:59:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.029+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22000cc50 con 0x7fb2300c36b0 2024-03-16T19:59:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.029+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2300cd290 con 0x7fb2300c36b0 2024-03-16T19:59:50.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.029+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2300c8760 con 0x7fb2300c36b0 2024-03-16T19:59:50.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.473+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb220013470 con 0x7fb2300c36b0 2024-03-16T19:59:50.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.473+0000 7fb21effd640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 0x7fb210069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:50.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.473+0000 7fb22dd9b640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 0x7fb210069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:50.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.473+0000 7fb22dd9b640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 0x7fb210069300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb230154650 tx=0x7fb224009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.501+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb220055e40 con 0x7fb2300c36b0 2024-03-16T19:59:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.501+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb2200520e0 con 0x7fb2300c36b0 2024-03-16T19:59:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.721+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb230068000 con 0x7fb2300c36b0 2024-03-16T19:59:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.745+0000 7fb21effd640 1 -- 172.21.15.32:0/3380066994 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fb220021090 con 0x7fb2300c36b0 2024-03-16T19:59:50.719 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:50.720 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 msgr2=0x7fb210069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 0x7fb210069300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb230154650 tx=0x7fb224009190 comp rx=0 tx=0).stop 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 msgr2=0x7fb2300cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb22000e8a0 tx=0x7fb22000eca0 comp rx=0 tx=0).stop 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 shutdown_connections 2024-03-16T19:59:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb210066e40 0x7fb210069300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb230151000 0x7fb23006b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2300c8980 0x7fb2301f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 --2- 172.21.15.32:0/3380066994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2300c36b0 0x7fb2300cc250 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 >> 172.21.15.32:0/3380066994 conn(0x7fb23006c0d0 msgr2=0x7fb23014e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 shutdown_connections 2024-03-16T19:59:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:50.749+0000 7fb234f0c640 1 -- 172.21.15.32:0/3380066994 wait complete. 2024-03-16T19:59:51.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:51.342 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:52.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.477+0000 7fbb0631e640 1 Processor -- start 2024-03-16T19:59:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.477+0000 7fbb0631e640 1 -- start start 2024-03-16T19:59:52.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb0631e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 0x7fbb0006be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:52.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb0631e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0006bca0 con 0x7fbb000b1ea0 2024-03-16T19:59:52.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb0631e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb00156cf0 con 0x7fbb00150710 2024-03-16T19:59:52.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb0631e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0006c500 con 0x7fbb00157180 2024-03-16T19:59:52.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb048bb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbb000b1ea0 0x7fbb00072a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50374/0 (socket says 172.21.15.32:50374) 2024-03-16T19:59:52.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbb048bb640 1 -- 172.21.15.32:0/39064771 learned_addr learned my addr 172.21.15.32:0/39064771 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:59:52.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.481+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/39064771 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 0x7fbb0006be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.485+0000 7fbafeffd640 1 -- 172.21.15.32:0/39064771 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618295183 0 0) 0x7fbb0006bca0 con 0x7fbb000b1ea0 2024-03-16T19:59:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.485+0000 7fbafeffd640 1 -- 172.21.15.32:0/39064771 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8004640 con 0x7fbb000b1ea0 2024-03-16T19:59:52.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.485+0000 7fbaff7fe640 1 -- 172.21.15.32:0/39064771 >> v1:172.21.15.59:6789/0 conn(0x7fbb00150710 legacy=0x7fbb0006d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:52.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.485+0000 7fbaff7fe640 1 -- 172.21.15.32:0/39064771 >> v1:172.21.15.32:6789/0 conn(0x7fbb000b1ea0 legacy=0x7fbb00072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:52.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.485+0000 7fbaff7fe640 1 -- 172.21.15.32:0/39064771 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0006c7a0 con 0x7fbb00157180 2024-03-16T19:59:52.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.489+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/39064771 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 0x7fbb0006be70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097f0 tx=0x7fbaf401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6e7d8efc30af24e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:52.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbafeffd640 1 -- 172.21.15.32:0/39064771 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4004230 con 0x7fbb00157180 2024-03-16T19:59:52.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbafeffd640 1 -- 172.21.15.32:0/39064771 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf40a7050 con 0x7fbb00157180 2024-03-16T19:59:52.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbafeffd640 1 -- 172.21.15.32:0/39064771 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003620 con 0x7fbb00157180 2024-03-16T19:59:52.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 -- 172.21.15.32:0/39064771 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 msgr2=0x7fbb0006be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:52.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 --2- 172.21.15.32:0/39064771 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 0x7fbb0006be70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097f0 tx=0x7fbaf401c4e0 comp rx=0 tx=0).stop 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 -- 172.21.15.32:0/39064771 shutdown_connections 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 --2- 172.21.15.32:0/39064771 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb00157180 0x7fbb0006be70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 -- 172.21.15.32:0/39064771 >> 172.21.15.32:0/39064771 conn(0x7fbb0006afc0 msgr2=0x7fbb0006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 -- 172.21.15.32:0/39064771 shutdown_connections 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.501+0000 7fbb0631e640 1 -- 172.21.15.32:0/39064771 wait complete. 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 Processor -- start 2024-03-16T19:59:52.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 -- start start 2024-03-16T19:59:52.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb000b1ea0 0x7fbb001f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:52.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 0x7fbb001f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:52.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 0x7fbb000c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:52.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000c3370 con 0x7fbb00150710 2024-03-16T19:59:52.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0014ad80 con 0x7fbb00157180 2024-03-16T19:59:52.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb0631e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001f12f0 con 0x7fbb000b1ea0 2024-03-16T19:59:52.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.505+0000 7fbb048bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 0x7fbb000c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.509+0000 7fbb048bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 0x7fbb000c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:59564/0 (socket says 172.21.15.32:59564) 2024-03-16T19:59:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.509+0000 7fbb048bb640 1 -- 172.21.15.32:0/166391954 learned_addr learned my addr 172.21.15.32:0/166391954 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.509+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 0x7fbb001f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:52.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.509+0000 7fbafffff640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb000b1ea0 0x7fbb001f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.513+0000 7fbaff7fe640 1 -- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb000b1ea0 msgr2=0x7fbb001f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.513+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb000b1ea0 0x7fbb001f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.513+0000 7fbaff7fe640 1 -- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 msgr2=0x7fbb000c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.513+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 0x7fbb000c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.513+0000 7fbaff7fe640 1 -- 172.21.15.32:0/166391954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf4009420 con 0x7fbb00150710 2024-03-16T19:59:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.521+0000 7fbaff7fe640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 0x7fbb001f0db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4004000 tx=0x7fbaf4019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40038f0 con 0x7fbb00150710 2024-03-16T19:59:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4003a50 con 0x7fbb00150710 2024-03-16T19:59:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003e00 con 0x7fbb00150710 2024-03-16T19:59:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0013e810 con 0x7fbb00150710 2024-03-16T19:59:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb001f46a0 con 0x7fbb00150710 2024-03-16T19:59:52.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.525+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb000b1a70 con 0x7fbb00150710 2024-03-16T19:59:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.565+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbaf40ab440 con 0x7fbb00150710 2024-03-16T19:59:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.565+0000 7fbafcff9640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 0x7fbaec069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.565+0000 7fbafffff640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 0x7fbaec069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.565+0000 7fbafffff640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 0x7fbaec069330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0001810 tx=0x7fbaf0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:52.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.569+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fbaf40f01c0 con 0x7fbb00150710 2024-03-16T19:59:52.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:52.901+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fbaf40ec460 con 0x7fbb00150710 2024-03-16T19:59:53.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.161+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb0006eca0 con 0x7fbb00150710 2024-03-16T19:59:53.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.237+0000 7fbafcff9640 1 -- 172.21.15.32:0/166391954 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fbaf40ba090 con 0x7fbb00150710 2024-03-16T19:59:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:53.212 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:53.212 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 msgr2=0x7fbaec069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 0x7fbaec069330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0001810 tx=0x7fbaf0009040 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 msgr2=0x7fbb001f0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 0x7fbb001f0db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4004000 tx=0x7fbaf4019280 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 shutdown_connections 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbaec066e70 0x7fbaec069330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb00157180 0x7fbb000c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb00150710 0x7fbb001f0db0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 --2- 172.21.15.32:0/166391954 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb000b1ea0 0x7fbb001f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 >> 172.21.15.32:0/166391954 conn(0x7fbb0006afc0 msgr2=0x7fbb0006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 shutdown_connections 2024-03-16T19:59:53.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:53.241+0000 7fbb0631e640 1 -- 172.21.15.32:0/166391954 wait complete. 2024-03-16T19:59:53.842 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:53.842 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:54.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 Processor -- start 2024-03-16T19:59:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 -- start start 2024-03-16T19:59:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 0x7f58e0070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e00c8590 con 0x7f58e00724a0 2024-03-16T19:59:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e0073590 con 0x7f58e00c7470 2024-03-16T19:59:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e7924640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00c3f00 con 0x7f58e0153fa0 2024-03-16T19:59:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e5ec1640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58e00724a0 0x7f58e0164770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50382/0 (socket says 172.21.15.32:50382) 2024-03-16T19:59:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e5ec1640 1 -- 172.21.15.32:0/403487258 learned_addr learned my addr 172.21.15.32:0/403487258 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T19:59:54.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.977+0000 7f58e56c0640 1 --2- 172.21.15.32:0/403487258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 0x7f58e0070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:54.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.981+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494780745 0 0) 0x7f58e00c8590 con 0x7f58e00724a0 2024-03-16T19:59:54.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.981+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc003530 con 0x7f58e00724a0 2024-03-16T19:59:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.981+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178755633 0 0) 0x7f58e0073590 con 0x7f58e00c7470 2024-03-16T19:59:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.981+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e00c8590 con 0x7f58e00c7470 2024-03-16T19:59:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.985+0000 7f58e56c0640 1 -- 172.21.15.32:0/403487258 >> v1:172.21.15.59:6789/0 conn(0x7f58e00c7470 legacy=0x7f58e015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.985+0000 7f58e56c0640 1 -- 172.21.15.32:0/403487258 >> v1:172.21.15.32:6789/0 conn(0x7f58e00724a0 legacy=0x7f58e0164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.985+0000 7f58e56c0640 1 -- 172.21.15.32:0/403487258 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e00c7890 con 0x7f58e0153fa0 2024-03-16T19:59:54.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.989+0000 7f58e56c0640 1 --2- 172.21.15.32:0/403487258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 0x7f58e0070840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0097f0 tx=0x7f58dc01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a966e714a0010461 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.997+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0a89d0 con 0x7f58e0153fa0 2024-03-16T19:59:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.997+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc0a8b30 con 0x7f58e0153fa0 2024-03-16T19:59:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:54.997+0000 7f58d7fff640 1 -- 172.21.15.32:0/403487258 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0b0440 con 0x7f58e0153fa0 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- 172.21.15.32:0/403487258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 msgr2=0x7f58e0070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 --2- 172.21.15.32:0/403487258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 0x7f58e0070840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0097f0 tx=0x7f58dc01ecd0 comp rx=0 tx=0).stop 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- 172.21.15.32:0/403487258 shutdown_connections 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 --2- 172.21.15.32:0/403487258 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e0153fa0 0x7f58e0070840 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- 172.21.15.32:0/403487258 >> 172.21.15.32:0/403487258 conn(0x7f58e006c0d0 msgr2=0x7f58e006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- 172.21.15.32:0/403487258 shutdown_connections 2024-03-16T19:59:54.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- 172.21.15.32:0/403487258 wait complete. 2024-03-16T19:59:54.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 Processor -- start 2024-03-16T19:59:54.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- start start 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e00724a0 0x7f58e0149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e00c7470 0x7f58e00c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0151320 con 0x7f58e0153fa0 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0140c30 con 0x7f58e00c7470 2024-03-16T19:59:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e7924640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00c4bf0 con 0x7f58e00724a0 2024-03-16T19:59:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e5ec1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e5ec1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58428/0 (socket says 172.21.15.32:58428) 2024-03-16T19:59:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e5ec1640 1 -- 172.21.15.32:0/2300963999 learned_addr learned my addr 172.21.15.32:0/2300963999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.001+0000 7f58e4ebf640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e00c7470 0x7f58e00c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:54.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e56c0640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e00724a0 0x7f58e0149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e5ec1640 1 -- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e00724a0 msgr2=0x7f58e0149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e5ec1640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e00724a0 0x7f58e0149740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e5ec1640 1 -- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e00c7470 msgr2=0x7f58e00c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e5ec1640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e00c7470 0x7f58e00c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.005+0000 7f58e5ec1640 1 -- 172.21.15.32:0/2300963999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc009420 con 0x7f58e0153fa0 2024-03-16T19:59:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.009+0000 7f58e5ec1640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f58d800cba0 tx=0x7f58d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:54.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.013+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d800ace0 con 0x7f58e0153fa0 2024-03-16T19:59:54.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.017+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d800ae40 con 0x7f58e0153fa0 2024-03-16T19:59:54.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.017+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e00c4910 con 0x7f58e0153fa0 2024-03-16T19:59:54.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.017+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8012620 con 0x7f58e0153fa0 2024-03-16T19:59:54.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.017+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e01f0160 con 0x7f58e0153fa0 2024-03-16T19:59:54.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.017+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e00c4da0 con 0x7f58e0153fa0 2024-03-16T19:59:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.309+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58d8007070 con 0x7f58e0153fa0 2024-03-16T19:59:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.309+0000 7f58d5ffb640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 0x7f58c4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.309+0000 7f58e56c0640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 0x7f58c4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:55.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.309+0000 7f58e56c0640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 0x7f58c4069350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0b2000 tx=0x7f58dc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:55.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.337+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f58d80558d0 con 0x7f58e0153fa0 2024-03-16T19:59:55.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.337+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f58d8051b70 con 0x7f58e0153fa0 2024-03-16T19:59:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.613+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e00c4da0 con 0x7f58e0153fa0 2024-03-16T19:59:55.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58d5ffb640 1 -- 172.21.15.32:0/2300963999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f58e00c4da0 con 0x7f58e0153fa0 2024-03-16T19:59:55.609 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:55.609 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 msgr2=0x7f58c4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 0x7f58c4069350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0b2000 tx=0x7f58dc0aa070 comp rx=0 tx=0).stop 2024-03-16T19:59:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 msgr2=0x7f58e00c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f58d800cba0 tx=0x7f58d800a430 comp rx=0 tx=0).stop 2024-03-16T19:59:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 shutdown_connections 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58c4066e90 0x7f58c4069350 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e0153fa0 0x7f58e00c55c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e00c7470 0x7f58e00c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 --2- 172.21.15.32:0/2300963999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e00724a0 0x7f58e0149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 >> 172.21.15.32:0/2300963999 conn(0x7f58e006c0d0 msgr2=0x7f58e01530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 shutdown_connections 2024-03-16T19:59:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:55.637+0000 7f58e7924640 1 -- 172.21.15.32:0/2300963999 wait complete. 2024-03-16T19:59:56.231 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:56.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:57.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:57.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 Processor -- start 2024-03-16T19:59:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 -- start start 2024-03-16T19:59:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6e6c0c6420 0x7f6e6c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c070760 con 0x7f6e6c0c6040 2024-03-16T19:59:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0731c0 con 0x7f6e6c0c6420 2024-03-16T19:59:57.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.365+0000 7f6e737cf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0734a0 con 0x7f6e6c158390 2024-03-16T19:59:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.369+0000 7f6e71d6c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.369+0000 7f6e70d6a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6e6c0c6420 0x7f6e6c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.369+0000 7f6e71d6c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:38134/0 (socket says 172.21.15.32:38134) 2024-03-16T19:59:57.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.369+0000 7f6e71d6c640 1 -- 172.21.15.32:0/7385997 learned_addr learned my addr 172.21.15.32:0/7385997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.373+0000 7f6e71d6c640 1 -- 172.21.15.32:0/7385997 >> v2:172.21.15.59:3300/0 conn(0x7f6e6c0c6420 msgr2=0x7f6e6c0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.373+0000 7f6e71d6c640 1 --2- 172.21.15.32:0/7385997 >> v2:172.21.15.59:3300/0 conn(0x7f6e6c0c6420 0x7f6e6c0c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.373+0000 7f6e71d6c640 1 -- 172.21.15.32:0/7385997 >> v1:172.21.15.59:6789/0 conn(0x7f6e6c0c6040 legacy=0x7f6e6c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.373+0000 7f6e71d6c640 1 -- 172.21.15.32:0/7385997 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c06fe30 con 0x7f6e6c158390 2024-03-16T19:59:57.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.381+0000 7f6e71d6c640 1 --2- 172.21.15.32:0/7385997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009e50 tx=0x7f6e6401c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d116e3bea72bb787 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e63fff640 1 -- 172.21.15.32:0/7385997 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640a5840 con 0x7f6e6c158390 2024-03-16T19:59:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e63fff640 1 -- 172.21.15.32:0/7385997 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e640a59a0 con 0x7f6e6c158390 2024-03-16T19:59:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e63fff640 1 -- 172.21.15.32:0/7385997 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640a5d30 con 0x7f6e6c158390 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 -- 172.21.15.32:0/7385997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 msgr2=0x7f6e6c1516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 --2- 172.21.15.32:0/7385997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009e50 tx=0x7f6e6401c9c0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 -- 172.21.15.32:0/7385997 shutdown_connections 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 --2- 172.21.15.32:0/7385997 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1516d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 --2- 172.21.15.32:0/7385997 >> v2:172.21.15.59:3300/0 conn(0x7f6e6c0c6420 0x7f6e6c0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 -- 172.21.15.32:0/7385997 >> 172.21.15.32:0/7385997 conn(0x7f6e6c06c0d0 msgr2=0x7f6e6c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 -- 172.21.15.32:0/7385997 shutdown_connections 2024-03-16T19:59:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.389+0000 7f6e737cf640 1 -- 172.21.15.32:0/7385997 wait complete. 2024-03-16T19:59:57.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 Processor -- start 2024-03-16T19:59:57.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 -- start start 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e6c0c6040 0x7f6e6c1f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1f3c80 con 0x7f6e6c0c6420 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c144110 con 0x7f6e6c0c6040 2024-03-16T19:59:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e737cf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1430f0 con 0x7f6e6c158390 2024-03-16T19:59:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e70d6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e70d6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58442/0 (socket says 172.21.15.32:58442) 2024-03-16T19:59:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e70d6a640 1 -- 172.21.15.32:0/1809187652 learned_addr learned my addr 172.21.15.32:0/1809187652 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e7156b640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e6c0c6040 0x7f6e6c1f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.393+0000 7f6e71d6c640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.397+0000 7f6e70d6a640 1 -- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 msgr2=0x7f6e6c1428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.397+0000 7f6e70d6a640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.397+0000 7f6e70d6a640 1 -- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e6c0c6040 msgr2=0x7f6e6c1f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.397+0000 7f6e70d6a640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e6c0c6040 0x7f6e6c1f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.397+0000 7f6e70d6a640 1 -- 172.21.15.32:0/1809187652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64009b30 con 0x7f6e6c0c6420 2024-03-16T19:59:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.401+0000 7f6e70d6a640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e900 tx=0x7f6e5c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c009690 con 0x7f6e6c0c6420 2024-03-16T19:59:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c00ca00 con 0x7f6e6c0c6420 2024-03-16T19:59:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c143b90 con 0x7f6e6c0c6420 2024-03-16T19:59:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c00cdb0 con 0x7f6e6c0c6420 2024-03-16T19:59:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c0c3d00 con 0x7f6e6c0c6420 2024-03-16T19:59:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.405+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c072c10 con 0x7f6e6c0c6420 2024-03-16T19:59:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.429+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6e5c013470 con 0x7f6e6c0c6420 2024-03-16T19:59:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.429+0000 7f6e61ffb640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 0x7f6e50069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.429+0000 7f6e7156b640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 0x7f6e50069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.433+0000 7f6e7156b640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 0x7f6e50069330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e68007fd0 tx=0x7f6e68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.717+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6e5c055fb0 con 0x7f6e6c0c6420 2024-03-16T19:59:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.717+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6e5c052250 con 0x7f6e6c0c6420 2024-03-16T19:59:57.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:57.997+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6c068000 con 0x7f6e6c0c6420 2024-03-16T19:59:58.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.069+0000 7f6e61ffb640 1 -- 172.21.15.32:0/1809187652 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f6e5c017090 con 0x7f6e6c0c6420 2024-03-16T19:59:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T19:59:58.044 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T19:59:58.044 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T19:59:58.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 msgr2=0x7f6e50069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:58.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 0x7f6e50069330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e68007fd0 tx=0x7f6e68009040 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 msgr2=0x7f6e6c142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e900 tx=0x7f6e5c00edc0 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 shutdown_connections 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e50066e70 0x7f6e50069330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e6c158390 0x7f6e6c1428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e6c0c6420 0x7f6e6c142360 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 --2- 172.21.15.32:0/1809187652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e6c0c6040 0x7f6e6c1f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 >> 172.21.15.32:0/1809187652 conn(0x7f6e6c06c0d0 msgr2=0x7f6e6c14edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 shutdown_connections 2024-03-16T19:59:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:58.073+0000 7f6e737cf640 1 -- 172.21.15.32:0/1809187652 wait complete. 2024-03-16T19:59:58.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T19:59:58.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T19:59:59.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:59:59.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.813+0000 7fc973e97640 1 Processor -- start 2024-03-16T19:59:59.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.813+0000 7fc973e97640 1 -- start start 2024-03-16T19:59:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc973e97640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc96c0c81e0 0x7fc96c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc973e97640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc973e97640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96c06c460 con 0x7fc96c1540c0 2024-03-16T19:59:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc973e97640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0c3f40 con 0x7fc96c0c81e0 2024-03-16T19:59:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc973e97640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c15a240 con 0x7fc96c0c3570 2024-03-16T19:59:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc972434640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc971432640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc96c0c81e0 0x7fc96c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc972434640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:38212/0 (socket says 172.21.15.32:38212) 2024-03-16T19:59:59.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.817+0000 7fc972434640 1 -- 172.21.15.32:0/2774581476 learned_addr learned my addr 172.21.15.32:0/2774581476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.821+0000 7fc972434640 1 -- 172.21.15.32:0/2774581476 >> v2:172.21.15.59:3300/0 conn(0x7fc96c0c81e0 msgr2=0x7fc96c0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.821+0000 7fc972434640 1 --2- 172.21.15.32:0/2774581476 >> v2:172.21.15.59:3300/0 conn(0x7fc96c0c81e0 0x7fc96c0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.821+0000 7fc972434640 1 -- 172.21.15.32:0/2774581476 >> v1:172.21.15.59:6789/0 conn(0x7fc96c1540c0 legacy=0x7fc96c15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:59:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.821+0000 7fc972434640 1 -- 172.21.15.32:0/2774581476 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c06cf20 con 0x7fc96c0c3570 2024-03-16T19:59:59.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.825+0000 7fc972434640 1 --2- 172.21.15.32:0/2774581476 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc964009e50 tx=0x7fc96401c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47d36f7e752d4657 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:59.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.829+0000 7fc970c31640 1 -- 172.21.15.32:0/2774581476 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640a5840 con 0x7fc96c0c3570 2024-03-16T19:59:59.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc970c31640 1 -- 172.21.15.32:0/2774581476 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9640a59a0 con 0x7fc96c0c3570 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- 172.21.15.32:0/2774581476 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 msgr2=0x7fc96c0c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 --2- 172.21.15.32:0/2774581476 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc964009e50 tx=0x7fc96401c9c0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- 172.21.15.32:0/2774581476 shutdown_connections 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 --2- 172.21.15.32:0/2774581476 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c0c3a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 --2- 172.21.15.32:0/2774581476 >> v2:172.21.15.59:3300/0 conn(0x7fc96c0c81e0 0x7fc96c0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- 172.21.15.32:0/2774581476 >> 172.21.15.32:0/2774581476 conn(0x7fc96c070f90 msgr2=0x7fc96c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- 172.21.15.32:0/2774581476 shutdown_connections 2024-03-16T19:59:59.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- 172.21.15.32:0/2774581476 wait complete. 2024-03-16T19:59:59.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 Processor -- start 2024-03-16T19:59:59.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.833+0000 7fc973e97640 1 -- start start 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc96c0c81e0 0x7fc96c06f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c147a60 con 0x7fc96c1540c0 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c14bcf0 con 0x7fc96c0c81e0 2024-03-16T19:59:59.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc973e97640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0cb810 con 0x7fc96c0c3570 2024-03-16T19:59:59.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc972434640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:59.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc971c33640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:59.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc972434640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57678/0 (socket says 172.21.15.32:57678) 2024-03-16T19:59:59.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc972434640 1 -- 172.21.15.32:0/3855315157 learned_addr learned my addr 172.21.15.32:0/3855315157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T19:59:59.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.837+0000 7fc971432640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc96c0c81e0 0x7fc96c06f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:59:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.841+0000 7fc972434640 1 -- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 msgr2=0x7fc96c147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.841+0000 7fc972434640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.841+0000 7fc972434640 1 -- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc96c0c81e0 msgr2=0x7fc96c06f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:59:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.841+0000 7fc972434640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc96c0c81e0 0x7fc96c06f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:59:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.841+0000 7fc972434640 1 -- 172.21.15.32:0/3855315157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc964009b30 con 0x7fc96c1540c0 2024-03-16T19:59:59.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.845+0000 7fc972434640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc96401b070 tx=0x7fc964019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:59:59.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964003ba0 con 0x7fc96c1540c0 2024-03-16T19:59:59.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964003d00 con 0x7fc96c1540c0 2024-03-16T19:59:59.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c0cbad0 con 0x7fc96c1540c0 2024-03-16T19:59:59.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964004260 con 0x7fc96c1540c0 2024-03-16T19:59:59.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c0cc090 con 0x7fc96c1540c0 2024-03-16T19:59:59.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T19:59:59.849+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c070390 con 0x7fc96c1540c0 2024-03-16T20:00:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.121+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc9640043c0 con 0x7fc96c1540c0 2024-03-16T20:00:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.125+0000 7fc9627fc640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 0x7fc950069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.125+0000 7fc971c33640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 0x7fc950069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.125+0000 7fc971c33640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 0x7fc950069350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc968007fd0 tx=0x7fc968009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.141+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc9640f1560 con 0x7fc96c1540c0 2024-03-16T20:00:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.141+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc9640ed800 con 0x7fc96c1540c0 2024-03-16T20:00:00.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.401+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc96c070390 con 0x7fc96c1540c0 2024-03-16T20:00:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc9627fc640 1 -- 172.21.15.32:0/3855315157 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fc9640f0ec0 con 0x7fc96c1540c0 2024-03-16T20:00:00.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:00.395 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 msgr2=0x7fc950069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 0x7fc950069350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc968007fd0 tx=0x7fc968009040 comp rx=0 tx=0).stop 2024-03-16T20:00:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 msgr2=0x7fc96c1f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc96401b070 tx=0x7fc964019760 comp rx=0 tx=0).stop 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 shutdown_connections 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc950066e90 0x7fc950069350 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc96c1540c0 0x7fc96c1f24c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc96c0c81e0 0x7fc96c06f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 --2- 172.21.15.32:0/3855315157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96c0c3570 0x7fc96c147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.425+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 >> 172.21.15.32:0/3855315157 conn(0x7fc96c070f90 msgr2=0x7fc96c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.429+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 shutdown_connections 2024-03-16T20:00:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:00.429+0000 7fc973e97640 1 -- 172.21.15.32:0/3855315157 wait complete. 2024-03-16T20:00:01.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:01.018 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:02.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.153+0000 7fe4a99f6640 1 Processor -- start 2024-03-16T20:00:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.153+0000 7fe4a99f6640 1 -- start start 2024-03-16T20:00:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a99f6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a99f6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe4a415b2a0 0x7fe4a415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a99f6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a4158c80 con 0x7fe4a40c97b0 2024-03-16T20:00:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a99f6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a40b0ad0 con 0x7fe4a415b2a0 2024-03-16T20:00:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a99f6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4157cb0 con 0x7fe4a4151ef0 2024-03-16T20:00:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a27fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe4a40c97b0 0x7fe4a40c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56302/0 (socket says 172.21.15.32:56302) 2024-03-16T20:00:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a27fc640 1 -- 172.21.15.32:0/1786325735 learned_addr learned my addr 172.21.15.32:0/1786325735 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/1786325735 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.157+0000 7fe4a37fe640 1 --2- 172.21.15.32:0/1786325735 >> v2:172.21.15.59:3300/0 conn(0x7fe4a415b2a0 0x7fe4a415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a1ffb640 1 -- 172.21.15.32:0/1786325735 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304232558 0 0) 0x7fe4a4158c80 con 0x7fe4a40c97b0 2024-03-16T20:00:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a1ffb640 1 -- 172.21.15.32:0/1786325735 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe48c003530 con 0x7fe4a40c97b0 2024-03-16T20:00:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a2ffd640 1 -- 172.21.15.32:0/1786325735 >> v2:172.21.15.59:3300/0 conn(0x7fe4a415b2a0 msgr2=0x7fe4a415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/1786325735 >> v2:172.21.15.59:3300/0 conn(0x7fe4a415b2a0 0x7fe4a415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a2ffd640 1 -- 172.21.15.32:0/1786325735 >> v1:172.21.15.32:6789/0 conn(0x7fe4a40c97b0 legacy=0x7fe4a40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.161+0000 7fe4a2ffd640 1 -- 172.21.15.32:0/1786325735 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a415bfc0 con 0x7fe4a4151ef0 2024-03-16T20:00:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.165+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/1786325735 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4940097c0 tx=0x7fe49401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad795c0f3f66d58 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:02.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.169+0000 7fe4a1ffb640 1 -- 172.21.15.32:0/1786325735 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940a8990 con 0x7fe4a4151ef0 2024-03-16T20:00:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a1ffb640 1 -- 172.21.15.32:0/1786325735 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940a8af0 con 0x7fe4a4151ef0 2024-03-16T20:00:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a1ffb640 1 -- 172.21.15.32:0/1786325735 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940a8e60 con 0x7fe4a4151ef0 2024-03-16T20:00:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 -- 172.21.15.32:0/1786325735 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 msgr2=0x7fe4a40c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/1786325735 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4940097c0 tx=0x7fe49401ec90 comp rx=0 tx=0).stop 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 -- 172.21.15.32:0/1786325735 shutdown_connections 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/1786325735 >> v2:172.21.15.59:3300/0 conn(0x7fe4a415b2a0 0x7fe4a415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/1786325735 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 -- 172.21.15.32:0/1786325735 >> 172.21.15.32:0/1786325735 conn(0x7fe4a4070f90 msgr2=0x7fe4a40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 -- 172.21.15.32:0/1786325735 shutdown_connections 2024-03-16T20:00:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.173+0000 7fe4a99f6640 1 -- 172.21.15.32:0/1786325735 wait complete. 2024-03-16T20:00:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 Processor -- start 2024-03-16T20:00:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 -- start start 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a40c97b0 0x7fe4a41f1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40ccfd0 con 0x7fe4a415b2a0 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41ed830 con 0x7fe4a40c97b0 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a99f6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41f0f40 con 0x7fe4a4151ef0 2024-03-16T20:00:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57686/0 (socket says 172.21.15.32:57686) 2024-03-16T20:00:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a37fe640 1 -- 172.21.15.32:0/2209210710 learned_addr learned my addr 172.21.15.32:0/2209210710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a27fc640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.177+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a40c97b0 0x7fe4a41f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.181+0000 7fe4a37fe640 1 -- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 msgr2=0x7fe4a40c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.181+0000 7fe4a37fe640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.181+0000 7fe4a37fe640 1 -- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a40c97b0 msgr2=0x7fe4a41f1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.181+0000 7fe4a37fe640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a40c97b0 0x7fe4a41f1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.181+0000 7fe4a37fe640 1 -- 172.21.15.32:0/2209210710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494009420 con 0x7fe4a415b2a0 2024-03-16T20:00:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.185+0000 7fe4a37fe640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00efc0 tx=0x7fe49c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.189+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c014480 con 0x7fe4a415b2a0 2024-03-16T20:00:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.189+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a40cc450 con 0x7fe4a415b2a0 2024-03-16T20:00:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.189+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a40cca10 con 0x7fe4a415b2a0 2024-03-16T20:00:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.189+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a41f1100 con 0x7fe4a415b2a0 2024-03-16T20:00:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.449+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c00f040 con 0x7fe4a415b2a0 2024-03-16T20:00:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.449+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c0147e0 con 0x7fe4a415b2a0 2024-03-16T20:00:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.449+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe49c014980 con 0x7fe4a415b2a0 2024-03-16T20:00:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.453+0000 7fe48bfff640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 0x7fe490069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:02.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.453+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 0x7fe490069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:02.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.453+0000 7fe4a2ffd640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 0x7fe490069300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe4940b2000 tx=0x7fe4940aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.465+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe49c056b70 con 0x7fe4a415b2a0 2024-03-16T20:00:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.465+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe49c052e10 con 0x7fe4a415b2a0 2024-03-16T20:00:02.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.689+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4a4068000 con 0x7fe4a415b2a0 2024-03-16T20:00:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe48bfff640 1 -- 172.21.15.32:0/2209210710 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fe49c0564d0 con 0x7fe4a415b2a0 2024-03-16T20:00:02.674 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:02.674 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 msgr2=0x7fe490069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 0x7fe490069300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe4940b2000 tx=0x7fe4940aa070 comp rx=0 tx=0).stop 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 msgr2=0x7fe4a40ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00efc0 tx=0x7fe49c00c490 comp rx=0 tx=0).stop 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 shutdown_connections 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe490066e40 0x7fe490069300 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe4940b2000 tx=0x7fe4940aa070 comp rx=0 tx=0).stop 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4a415b2a0 0x7fe4a40ca120 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe4a4151ef0 0x7fe4a40c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 --2- 172.21.15.32:0/2209210710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a40c97b0 0x7fe4a41f1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 >> 172.21.15.32:0/2209210710 conn(0x7fe4a4070f90 msgr2=0x7fe4a40c3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 shutdown_connections 2024-03-16T20:00:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:02.705+0000 7fe4a99f6640 1 -- 172.21.15.32:0/2209210710 wait complete. 2024-03-16T20:00:03.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:03.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:04.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:04.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 Processor -- start 2024-03-16T20:00:04.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 -- start start 2024-03-16T20:00:04.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c3e30 0x7fad58150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad580c4210 0x7fad58151050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580c3940 con 0x7fad58072ed0 2024-03-16T20:00:04.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad580c7ca0 con 0x7fad580c4210 2024-03-16T20:00:04.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.441+0000 7fad5f431640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad58072cc0 con 0x7fad580c3e30 2024-03-16T20:00:04.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.445+0000 7fad5d9ce640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fad58072ed0 0x7fad580c80f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56310/0 (socket says 172.21.15.32:56310) 2024-03-16T20:00:04.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.445+0000 7fad5d9ce640 1 -- 172.21.15.32:0/3487423496 learned_addr learned my addr 172.21.15.32:0/3487423496 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:04.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.445+0000 7fad5c9cc640 1 --2- 172.21.15.32:0/3487423496 >> v2:172.21.15.59:3300/0 conn(0x7fad580c4210 0x7fad58151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:04.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.445+0000 7fad4ffff640 1 -- 172.21.15.32:0/3487423496 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860725162 0 0) 0x7fad580c3940 con 0x7fad58072ed0 2024-03-16T20:00:04.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.445+0000 7fad4ffff640 1 -- 172.21.15.32:0/3487423496 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44003530 con 0x7fad58072ed0 2024-03-16T20:00:04.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.449+0000 7fad5c9cc640 1 -- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c3e30 msgr2=0x7fad58150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:04.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.449+0000 7fad5c9cc640 1 --2- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c3e30 0x7fad58150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.449+0000 7fad5c9cc640 1 -- 172.21.15.32:0/3487423496 >> v1:172.21.15.32:6789/0 conn(0x7fad58072ed0 legacy=0x7fad580c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:04.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.449+0000 7fad5c9cc640 1 -- 172.21.15.32:0/3487423496 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580c7780 con 0x7fad580c4210 2024-03-16T20:00:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.453+0000 7fad5c9cc640 1 --2- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad580c4210 0x7fad58151050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad480097c0 tx=0x7fad4801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=721bb8b72b8ac730 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad4ffff640 1 -- 172.21.15.32:0/3487423496 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48004230 con 0x7fad580c4210 2024-03-16T20:00:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad4ffff640 1 -- 172.21.15.32:0/3487423496 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad480a7050 con 0x7fad580c4210 2024-03-16T20:00:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad4ffff640 1 -- 172.21.15.32:0/3487423496 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48003620 con 0x7fad580c4210 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 -- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad580c4210 msgr2=0x7fad58151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 --2- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad580c4210 0x7fad58151050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad480097c0 tx=0x7fad4801c4a0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 -- 172.21.15.32:0/3487423496 shutdown_connections 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 --2- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad580c4210 0x7fad58151050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 --2- 172.21.15.32:0/3487423496 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c3e30 0x7fad58150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 -- 172.21.15.32:0/3487423496 >> 172.21.15.32:0/3487423496 conn(0x7fad5806e6c0 msgr2=0x7fad5806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 -- 172.21.15.32:0/3487423496 shutdown_connections 2024-03-16T20:00:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.461+0000 7fad5f431640 1 -- 172.21.15.32:0/3487423496 wait complete. 2024-03-16T20:00:04.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 Processor -- start 2024-03-16T20:00:04.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 -- start start 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580c3e30 0x7fad58156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c4210 0x7fad581f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58147990 con 0x7fad580c3e30 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58158e90 con 0x7fad58072ed0 2024-03-16T20:00:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5f431640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad58148760 con 0x7fad580c4210 2024-03-16T20:00:04.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5d1cd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:04.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5d1cd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:33736/0 (socket says 172.21.15.32:33736) 2024-03-16T20:00:04.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.465+0000 7fad5d1cd640 1 -- 172.21.15.32:0/3562032458 learned_addr learned my addr 172.21.15.32:0/3562032458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:04.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.469+0000 7fad5d9ce640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c4210 0x7fad581f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:04.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.473+0000 7fad5d1cd640 1 -- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c4210 msgr2=0x7fad581f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:04.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.473+0000 7fad5d1cd640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c4210 0x7fad581f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.473+0000 7fad5d1cd640 1 -- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580c3e30 msgr2=0x7fad58156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:04.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.473+0000 7fad5d1cd640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580c3e30 0x7fad58156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:04.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.473+0000 7fad5d1cd640 1 -- 172.21.15.32:0/3562032458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009420 con 0x7fad58072ed0 2024-03-16T20:00:04.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.477+0000 7fad5d1cd640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fad5400b550 tx=0x7fad5400ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:04.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.485+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54010870 con 0x7fad58072ed0 2024-03-16T20:00:04.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.485+0000 7fad5f431640 1 -- 172.21.15.32:0/3562032458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad58158bf0 con 0x7fad58072ed0 2024-03-16T20:00:04.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.485+0000 7fad5f431640 1 -- 172.21.15.32:0/3562032458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad58148110 con 0x7fad58072ed0 2024-03-16T20:00:04.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.485+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad380052a0 con 0x7fad58072ed0 2024-03-16T20:00:04.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.485+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540109d0 con 0x7fad58072ed0 2024-03-16T20:00:04.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.489+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54010d40 con 0x7fad58072ed0 2024-03-16T20:00:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.525+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fad54023480 con 0x7fad58072ed0 2024-03-16T20:00:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.525+0000 7fad4dffb640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 0x7fad3c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.525+0000 7fad5c9cc640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 0x7fad3c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.529+0000 7fad5c9cc640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 0x7fad3c069330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fad480ad000 tx=0x7fad48004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:04.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.825+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fad540559e0 con 0x7fad58072ed0 2024-03-16T20:00:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:04.849+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fad54051c80 con 0x7fad58072ed0 2024-03-16T20:00:05.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.061+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad38005900 con 0x7fad58072ed0 2024-03-16T20:00:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.185+0000 7fad4dffb640 1 -- 172.21.15.32:0/3562032458 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fad38005900 con 0x7fad58072ed0 2024-03-16T20:00:05.158 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:05.158 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 msgr2=0x7fad3c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 0x7fad3c069330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fad480ad000 tx=0x7fad48004740 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 msgr2=0x7fad581585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fad5400b550 tx=0x7fad5400ba10 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 shutdown_connections 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fad3c066e70 0x7fad3c069330 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fad580c4210 0x7fad581f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580c3e30 0x7fad58156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 --2- 172.21.15.32:0/3562032458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad58072ed0 0x7fad581585d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 >> 172.21.15.32:0/3562032458 conn(0x7fad5806e6c0 msgr2=0x7fad5806b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 shutdown_connections 2024-03-16T20:00:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:05.189+0000 7fad437fe640 1 -- 172.21.15.32:0/3562032458 wait complete. 2024-03-16T20:00:05.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:05.786 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:06.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 Processor -- start 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 -- start start 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5340c81e0 0x7fb5340c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb534158c80 con 0x7fb53415b2a0 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53415bfc0 con 0x7fb534151ef0 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb53a0d5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5340b0ad0 con 0x7fb5340c81e0 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb533fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb53415b2a0 0x7fb53415b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56324/0 (socket says 172.21.15.32:56324) 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb533fff640 1 -- 172.21.15.32:0/1940226286 learned_addr learned my addr 172.21.15.32:0/1940226286 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.925+0000 7fb532ffd640 1 --2- 172.21.15.32:0/1940226286 >> v2:172.21.15.59:3300/0 conn(0x7fb5340c81e0 0x7fb5340c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.929+0000 7fb5327fc640 1 -- 172.21.15.32:0/1940226286 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429765355 0 0) 0x7fb534158c80 con 0x7fb53415b2a0 2024-03-16T20:00:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.929+0000 7fb5327fc640 1 -- 172.21.15.32:0/1940226286 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb51c003530 con 0x7fb53415b2a0 2024-03-16T20:00:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.933+0000 7fb532ffd640 1 -- 172.21.15.32:0/1940226286 >> v1:172.21.15.59:6789/0 conn(0x7fb534151ef0 legacy=0x7fb5340c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.933+0000 7fb532ffd640 1 -- 172.21.15.32:0/1940226286 >> v1:172.21.15.32:6789/0 conn(0x7fb53415b2a0 legacy=0x7fb53415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.933+0000 7fb532ffd640 1 -- 172.21.15.32:0/1940226286 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534157ec0 con 0x7fb5340c81e0 2024-03-16T20:00:06.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.941+0000 7fb532ffd640 1 --2- 172.21.15.32:0/1940226286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb5340c8630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097c0 tx=0x7fb52801ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82b15efe8ae7168 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:06.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb5327fc640 1 -- 172.21.15.32:0/1940226286 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280a8950 con 0x7fb5340c81e0 2024-03-16T20:00:06.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb5327fc640 1 -- 172.21.15.32:0/1940226286 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5280a8ab0 con 0x7fb5340c81e0 2024-03-16T20:00:06.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb5327fc640 1 -- 172.21.15.32:0/1940226286 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280a8e40 con 0x7fb5340c81e0 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- 172.21.15.32:0/1940226286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 msgr2=0x7fb5340c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/1940226286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb5340c8630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097c0 tx=0x7fb52801ec50 comp rx=0 tx=0).stop 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- 172.21.15.32:0/1940226286 shutdown_connections 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/1940226286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb5340c8630 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- 172.21.15.32:0/1940226286 >> 172.21.15.32:0/1940226286 conn(0x7fb534070f90 msgr2=0x7fb5340713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- 172.21.15.32:0/1940226286 shutdown_connections 2024-03-16T20:00:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- 172.21.15.32:0/1940226286 wait complete. 2024-03-16T20:00:06.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 Processor -- start 2024-03-16T20:00:06.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.953+0000 7fb53a0d5640 1 -- start start 2024-03-16T20:00:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb53414de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb534151ef0 0x7fb534155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5341eddf0 con 0x7fb53415b2a0 2024-03-16T20:00:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5341ed7c0 con 0x7fb5340c81e0 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb53a0d5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53414d500 con 0x7fb534151ef0 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb533fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb533fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57722/0 (socket says 172.21.15.32:57722) 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb533fff640 1 -- 172.21.15.32:0/2976043352 learned_addr learned my addr 172.21.15.32:0/2976043352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb5337fe640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb53414de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.957+0000 7fb532ffd640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb534151ef0 0x7fb534155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:06.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 -- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb534151ef0 msgr2=0x7fb534155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:06.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb534151ef0 0x7fb534155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:06.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 -- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 msgr2=0x7fb53414de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:06.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb53414de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:06.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 -- 172.21.15.32:0/2976043352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528009420 con 0x7fb53415b2a0 2024-03-16T20:00:06.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.961+0000 7fb533fff640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00caa0 tx=0x7fb52c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00ac00 con 0x7fb53415b2a0 2024-03-16T20:00:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c00ad60 con 0x7fb53415b2a0 2024-03-16T20:00:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53414d7e0 con 0x7fb53415b2a0 2024-03-16T20:00:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c012620 con 0x7fb53415b2a0 2024-03-16T20:00:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5341eda80 con 0x7fb53415b2a0 2024-03-16T20:00:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.969+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5341edc10 con 0x7fb53415b2a0 2024-03-16T20:00:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.993+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb52c007070 con 0x7fb53415b2a0 2024-03-16T20:00:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.993+0000 7fb51bfff640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 0x7fb520069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:06.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.993+0000 7fb5337fe640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 0x7fb520069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:06.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:06.993+0000 7fb5337fe640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 0x7fb520069330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb524007fd0 tx=0x7fb524009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:07.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.265+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb52c0559b0 con 0x7fb53415b2a0 2024-03-16T20:00:07.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.265+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb52c051c50 con 0x7fb53415b2a0 2024-03-16T20:00:07.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.461+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb534068000 con 0x7fb53415b2a0 2024-03-16T20:00:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb51bfff640 1 -- 172.21.15.32:0/2976043352 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fb52c017090 con 0x7fb53415b2a0 2024-03-16T20:00:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:07.450 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 msgr2=0x7fb520069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 0x7fb520069330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb524007fd0 tx=0x7fb524009040 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 msgr2=0x7fb5341f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00caa0 tx=0x7fb52c00cf60 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 shutdown_connections 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb520066e70 0x7fb520069330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415b2a0 0x7fb5341f1740 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb534151ef0 0x7fb534155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 --2- 172.21.15.32:0/2976043352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5340c81e0 0x7fb53414de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 >> 172.21.15.32:0/2976043352 conn(0x7fb534070f90 msgr2=0x7fb53415b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 shutdown_connections 2024-03-16T20:00:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:07.481+0000 7fb53a0d5640 1 -- 172.21.15.32:0/2976043352 wait complete. 2024-03-16T20:00:08.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:08.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:09.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:09.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.217+0000 7f2d52c86640 1 Processor -- start 2024-03-16T20:00:09.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.217+0000 7f2d52c86640 1 -- start start 2024-03-16T20:00:09.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d52c86640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c157180 0x7f2d4c06be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d52c86640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2d4c0b1ea0 0x7f2d4c072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d52c86640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4c06bca0 con 0x7f2d4c150710 2024-03-16T20:00:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d52c86640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4c156cf0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d52c86640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c06c500 con 0x7f2d4c157180 2024-03-16T20:00:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d51223640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2d4c0b1ea0 0x7f2d4c072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d50a22640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2d4c150710 0x7f2d4c06d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:55840/0 (socket says 172.21.15.32:55840) 2024-03-16T20:00:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d50a22640 1 -- 172.21.15.32:0/1563173123 learned_addr learned my addr 172.21.15.32:0/1563173123 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.221+0000 7f2d4bfff640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c157180 0x7f2d4c06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.225+0000 7f2d51223640 1 -- 172.21.15.32:0/1563173123 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c157180 msgr2=0x7f2d4c06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.225+0000 7f2d51223640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c157180 0x7f2d4c06be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.225+0000 7f2d51223640 1 -- 172.21.15.32:0/1563173123 >> v1:172.21.15.59:6789/0 conn(0x7f2d4c150710 legacy=0x7f2d4c06d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.225+0000 7f2d51223640 1 -- 172.21.15.32:0/1563173123 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c06c7a0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.229+0000 7f2d51223640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c072a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b35c53901ac2d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d4b7fe640 1 -- 172.21.15.32:0/1563173123 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440a5920 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d4b7fe640 1 -- 172.21.15.32:0/1563173123 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d440a5a80 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d4b7fe640 1 -- 172.21.15.32:0/1563173123 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440a5e10 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 -- 172.21.15.32:0/1563173123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 msgr2=0x7f2d4c072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c072a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4401c970 comp rx=0 tx=0).stop 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 -- 172.21.15.32:0/1563173123 shutdown_connections 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c072a10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1563173123 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c157180 0x7f2d4c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.233+0000 7f2d52c86640 1 -- 172.21.15.32:0/1563173123 >> 172.21.15.32:0/1563173123 conn(0x7f2d4c06afc0 msgr2=0x7f2d4c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- 172.21.15.32:0/1563173123 shutdown_connections 2024-03-16T20:00:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- 172.21.15.32:0/1563173123 wait complete. 2024-03-16T20:00:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 Processor -- start 2024-03-16T20:00:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- start start 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c150710 0x7f2d4c0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d4c157180 0x7f2d4c14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c0c7c00 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c14ff40 con 0x7f2d4c157180 2024-03-16T20:00:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d52c86640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c14d520 con 0x7f2d4c150710 2024-03-16T20:00:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d50a22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d50a22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55332/0 (socket says 172.21.15.32:55332) 2024-03-16T20:00:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d50a22640 1 -- 172.21.15.32:0/1540603433 learned_addr learned my addr 172.21.15.32:0/1540603433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d51223640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d4c157180 0x7f2d4c14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.237+0000 7f2d4bfff640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c150710 0x7f2d4c0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.241+0000 7f2d50a22640 1 -- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c150710 msgr2=0x7f2d4c0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.241+0000 7f2d50a22640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c150710 0x7f2d4c0cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.241+0000 7f2d50a22640 1 -- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d4c157180 msgr2=0x7f2d4c14da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.241+0000 7f2d50a22640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d4c157180 0x7f2d4c14da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.241+0000 7f2d50a22640 1 -- 172.21.15.32:0/1540603433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44009a70 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.245+0000 7f2d50a22640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c500 tx=0x7f2d4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40005cf0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d40005e50 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4c0c7080 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400072f0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4c0c7640 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.249+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4c14d380 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.285+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2d40007450 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.285+0000 7f2d497fa640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 0x7f2d30069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.285+0000 7f2d4bfff640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 0x7f2d30069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.285+0000 7f2d4bfff640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 0x7f2d30069350 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0c6af0 tx=0x7f2d38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:09.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.589+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2d40054e40 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.589+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2d400510e0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.793+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d4c068000 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.817+0000 7f2d497fa640 1 -- 172.21.15.32:0/1540603433 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f2d400547a0 con 0x7f2d4c0b1ea0 2024-03-16T20:00:09.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:09.789 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 msgr2=0x7f2d30069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 0x7f2d30069350 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0c6af0 tx=0x7f2d38008040 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 msgr2=0x7f2d4c14f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c500 tx=0x7f2d4000c9c0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 shutdown_connections 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2d30066e90 0x7f2d30069350 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d4c157180 0x7f2d4c14da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2d4c150710 0x7f2d4c0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 --2- 172.21.15.32:0/1540603433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d4c0b1ea0 0x7f2d4c14f4d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 >> 172.21.15.32:0/1540603433 conn(0x7f2d4c06afc0 msgr2=0x7f2d4c155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 shutdown_connections 2024-03-16T20:00:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:09.821+0000 7f2d52c86640 1 -- 172.21.15.32:0/1540603433 wait complete. 2024-03-16T20:00:10.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:10.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:11.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 Processor -- start 2024-03-16T20:00:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 -- start start 2024-03-16T20:00:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 0x7f443006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443006bae0 con 0x7f4430154350 2024-03-16T20:00:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443015a540 con 0x7f4430072a10 2024-03-16T20:00:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4438018640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300c7fd0 con 0x7f44300c2d70 2024-03-16T20:00:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4435db4640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4430154350 0x7f443015a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44440/0 (socket says 172.21.15.32:44440) 2024-03-16T20:00:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.557+0000 7f4435db4640 1 -- 172.21.15.32:0/3258358895 learned_addr learned my addr 172.21.15.32:0/3258358895 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:11.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.561+0000 7f44355b3640 1 --2- 172.21.15.32:0/3258358895 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 0x7f443006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:11.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979395329 0 0) 0x7f443006bae0 con 0x7f4430154350 2024-03-16T20:00:11.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f441c003530 con 0x7f4430154350 2024-03-16T20:00:11.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553051745 0 0) 0x7f443015a540 con 0x7f4430072a10 2024-03-16T20:00:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f443006bae0 con 0x7f4430072a10 2024-03-16T20:00:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f44355b3640 1 -- 172.21.15.32:0/3258358895 >> v1:172.21.15.59:6789/0 conn(0x7f4430072a10 legacy=0x7f44301649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f44355b3640 1 -- 172.21.15.32:0/3258358895 >> v1:172.21.15.32:6789/0 conn(0x7f4430154350 legacy=0x7f443015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.565+0000 7f44355b3640 1 -- 172.21.15.32:0/3258358895 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443006bd10 con 0x7f44300c2d70 2024-03-16T20:00:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.573+0000 7f44355b3640 1 --2- 172.21.15.32:0/3258358895 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 0x7f443006c250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44200097f0 tx=0x7f442001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b9ea31ca98621d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.577+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200ab600 con 0x7f44300c2d70 2024-03-16T20:00:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4434db2640 1 -- 172.21.15.32:0/3258358895 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44200a7050 con 0x7f44300c2d70 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 -- 172.21.15.32:0/3258358895 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 msgr2=0x7f443006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 --2- 172.21.15.32:0/3258358895 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 0x7f443006c250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44200097f0 tx=0x7f442001c450 comp rx=0 tx=0).stop 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 -- 172.21.15.32:0/3258358895 shutdown_connections 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 --2- 172.21.15.32:0/3258358895 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44300c2d70 0x7f443006c250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 -- 172.21.15.32:0/3258358895 >> 172.21.15.32:0/3258358895 conn(0x7f443006afc0 msgr2=0x7f443006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 -- 172.21.15.32:0/3258358895 shutdown_connections 2024-03-16T20:00:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.581+0000 7f4438018640 1 -- 172.21.15.32:0/3258358895 wait complete. 2024-03-16T20:00:11.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 Processor -- start 2024-03-16T20:00:11.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 -- start start 2024-03-16T20:00:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4430072a10 0x7f4430149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4430154350 0x7f443014cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443015c140 con 0x7f44300c2d70 2024-03-16T20:00:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4430153380 con 0x7f4430072a10 2024-03-16T20:00:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4438018640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443014d590 con 0x7f4430154350 2024-03-16T20:00:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f44355b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f4435db4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4430072a10 0x7f4430149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f44355b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55336/0 (socket says 172.21.15.32:55336) 2024-03-16T20:00:11.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f44355b3640 1 -- 172.21.15.32:0/1977797837 learned_addr learned my addr 172.21.15.32:0/1977797837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:11.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.585+0000 7f44365b5640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4430154350 0x7f443014cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.589+0000 7f44355b3640 1 -- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4430154350 msgr2=0x7f443014cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.589+0000 7f44355b3640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4430154350 0x7f443014cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.589+0000 7f44355b3640 1 -- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4430072a10 msgr2=0x7f4430149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.589+0000 7f44355b3640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4430072a10 0x7f4430149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.589+0000 7f44355b3640 1 -- 172.21.15.32:0/1977797837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420009420 con 0x7f44300c2d70 2024-03-16T20:00:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.593+0000 7f44355b3640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c10 tx=0x7f44200191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:11.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420003ab0 con 0x7f44300c2d70 2024-03-16T20:00:11.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4420003c10 con 0x7f44300c2d70 2024-03-16T20:00:11.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430152d70 con 0x7f44300c2d70 2024-03-16T20:00:11.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200b4430 con 0x7f44300c2d70 2024-03-16T20:00:11.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443015bb80 con 0x7f44300c2d70 2024-03-16T20:00:11.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.597+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4430152bb0 con 0x7f44300c2d70 2024-03-16T20:00:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.621+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f44200b2040 con 0x7f44300c2d70 2024-03-16T20:00:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.621+0000 7f44267fc640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 0x7f4414069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.621+0000 7f4435db4640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 0x7f4414069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.621+0000 7f4435db4640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 0x7f4414069350 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f442c000f70 tx=0x7f442c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:11.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.629+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f44200efea0 con 0x7f44300c2d70 2024-03-16T20:00:11.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:00:11.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:00:11.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:11.933+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f44200ec140 con 0x7f44300c2d70 2024-03-16T20:00:12.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.149+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44300674e0 con 0x7f44300c2d70 2024-03-16T20:00:12.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.173+0000 7f44267fc640 1 -- 172.21.15.32:0/1977797837 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f44200b0070 con 0x7f44300c2d70 2024-03-16T20:00:12.144 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:12.144 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:12.145 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 msgr2=0x7f4414069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 0x7f4414069350 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f442c000f70 tx=0x7f442c009040 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 msgr2=0x7f443014ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c10 tx=0x7f44200191f0 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 shutdown_connections 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4414066e90 0x7f4414069350 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4430154350 0x7f443014cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44300c2d70 0x7f443014ca50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 --2- 172.21.15.32:0/1977797837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4430072a10 0x7f4430149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 >> 172.21.15.32:0/1977797837 conn(0x7f443006afc0 msgr2=0x7f4430159980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 shutdown_connections 2024-03-16T20:00:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:12.177+0000 7f4438018640 1 -- 172.21.15.32:0/1977797837 wait complete. 2024-03-16T20:00:12.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:12.767 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:13.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 Processor -- start 2024-03-16T20:00:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 -- start start 2024-03-16T20:00:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45d8072c70 0x7f45d80c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 0x7f45d80c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d806e6a0 con 0x7f45d806bae0 2024-03-16T20:00:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d8156f20 con 0x7f45d8072c70 2024-03-16T20:00:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.905+0000 7f45dcd1c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d806c250 con 0x7f45d8073050 2024-03-16T20:00:13.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.909+0000 7f45d659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45d8072c70 0x7f45d80c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.909+0000 7f45d5d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 0x7f45d80c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.909+0000 7f45d6d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f45d806bae0 0x7f45d8150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:55858/0 (socket says 172.21.15.32:55858) 2024-03-16T20:00:13.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.909+0000 7f45d6d9d640 1 -- 172.21.15.32:0/2166064411 learned_addr learned my addr 172.21.15.32:0/2166064411 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.917+0000 7f45d5d9b640 1 -- 172.21.15.32:0/2166064411 >> v2:172.21.15.59:3300/0 conn(0x7f45d8072c70 msgr2=0x7f45d80c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.917+0000 7f45d5d9b640 1 --2- 172.21.15.32:0/2166064411 >> v2:172.21.15.59:3300/0 conn(0x7f45d8072c70 0x7f45d80c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.917+0000 7f45d5d9b640 1 -- 172.21.15.32:0/2166064411 >> v1:172.21.15.59:6789/0 conn(0x7f45d806bae0 legacy=0x7f45d8150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.917+0000 7f45d5d9b640 1 -- 172.21.15.32:0/2166064411 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d806c510 con 0x7f45d8073050 2024-03-16T20:00:13.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.925+0000 7f45d5d9b640 1 --2- 172.21.15.32:0/2166064411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 0x7f45d80c14d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0097c0 tx=0x7f45cc01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=885a10df80ad9683 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.933+0000 7f45d559a640 1 -- 172.21.15.32:0/2166064411 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0aa5f0 con 0x7f45d8073050 2024-03-16T20:00:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d559a640 1 -- 172.21.15.32:0/2166064411 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc0aa750 con 0x7f45d8073050 2024-03-16T20:00:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d559a640 1 -- 172.21.15.32:0/2166064411 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0aaae0 con 0x7f45d8073050 2024-03-16T20:00:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- 172.21.15.32:0/2166064411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 msgr2=0x7f45d80c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/2166064411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 0x7f45d80c14d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0097c0 tx=0x7f45cc01c470 comp rx=0 tx=0).stop 2024-03-16T20:00:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- 172.21.15.32:0/2166064411 shutdown_connections 2024-03-16T20:00:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/2166064411 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d8073050 0x7f45d80c14d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/2166064411 >> v2:172.21.15.59:3300/0 conn(0x7f45d8072c70 0x7f45d80c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- 172.21.15.32:0/2166064411 >> 172.21.15.32:0/2166064411 conn(0x7f45d806afc0 msgr2=0x7f45d806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- 172.21.15.32:0/2166064411 shutdown_connections 2024-03-16T20:00:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- 172.21.15.32:0/2166064411 wait complete. 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 Processor -- start 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- start start 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d806bae0 0x7f45d8156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d8073050 0x7f45d814f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d81557d0 con 0x7f45d8072c70 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d81564a0 con 0x7f45d8073050 2024-03-16T20:00:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45dcd1c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d8141fd0 con 0x7f45d806bae0 2024-03-16T20:00:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d8073050 0x7f45d814f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55338/0 (socket says 172.21.15.32:55338) 2024-03-16T20:00:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.937+0000 7f45d5d9b640 1 -- 172.21.15.32:0/1840804373 learned_addr learned my addr 172.21.15.32:0/1840804373 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:13.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d659c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d806bae0 0x7f45d8156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d5d9b640 1 -- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d806bae0 msgr2=0x7f45d8156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d5d9b640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d806bae0 0x7f45d8156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d5d9b640 1 -- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d8073050 msgr2=0x7f45d814f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d5d9b640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d8073050 0x7f45d814f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.941+0000 7f45d5d9b640 1 -- 172.21.15.32:0/1840804373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc009420 con 0x7f45d8072c70 2024-03-16T20:00:13.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.945+0000 7f45d5d9b640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0ac040 tx=0x7f45cc01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.949+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0be610 con 0x7f45d8072c70 2024-03-16T20:00:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.949+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc0be790 con 0x7f45d8072c70 2024-03-16T20:00:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.949+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0beb40 con 0x7f45d8072c70 2024-03-16T20:00:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.949+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d80cbb70 con 0x7f45d8072c70 2024-03-16T20:00:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.949+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d80c3460 con 0x7f45d8072c70 2024-03-16T20:00:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.953+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d806d050 con 0x7f45d8072c70 2024-03-16T20:00:13.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.989+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f45cc0b4400 con 0x7f45d8072c70 2024-03-16T20:00:13.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.989+0000 7f45c6ffd640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 0x7f45b8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:13.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.989+0000 7f45d659c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 0x7f45b8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:13.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.989+0000 7f45d659c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 0x7f45b8069380 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009b60 tx=0x7f45c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:13.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:13.993+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f45cc0a7070 con 0x7f45d8072c70 2024-03-16T20:00:14.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.269+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f45cc0ec990 con 0x7f45d8072c70 2024-03-16T20:00:14.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.481+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45d8156610 con 0x7f45d8072c70 2024-03-16T20:00:14.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45c6ffd640 1 -- 172.21.15.32:0/1840804373 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f45cc0f0050 con 0x7f45d8072c70 2024-03-16T20:00:14.467 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:14.468 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:14.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 msgr2=0x7f45b8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 0x7f45b8069380 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009b60 tx=0x7f45c8009190 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 msgr2=0x7f45d80c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0ac040 tx=0x7f45cc01acb0 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 shutdown_connections 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f45b8066ec0 0x7f45b8069380 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d8073050 0x7f45d814f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45d8072c70 0x7f45d80c9ce0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 --2- 172.21.15.32:0/1840804373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d806bae0 0x7f45d8156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 >> 172.21.15.32:0/1840804373 conn(0x7f45d806afc0 msgr2=0x7f45d80c1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 shutdown_connections 2024-03-16T20:00:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:14.497+0000 7f45dcd1c640 1 -- 172.21.15.32:0/1840804373 wait complete. 2024-03-16T20:00:15.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:15.089 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:16.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:16.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.225+0000 7f877a9f3640 1 Processor -- start 2024-03-16T20:00:16.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.225+0000 7f877a9f3640 1 -- start start 2024-03-16T20:00:16.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f877a9f3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 0x7f87740c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f877a9f3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87740c97b0 0x7f87740c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f877a9f3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8774158c80 con 0x7f87740c8720 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f877a9f3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87740b0ad0 con 0x7f87740c97b0 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f877a9f3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8774157cb0 con 0x7f8774151ef0 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f8778f90640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f87740c8720 0x7f877415b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44452/0 (socket says 172.21.15.32:44452) 2024-03-16T20:00:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f8778f90640 1 -- 172.21.15.32:0/3205650112 learned_addr learned my addr 172.21.15.32:0/3205650112 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:16.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f87737fe640 1 --2- 172.21.15.32:0/3205650112 >> v2:172.21.15.59:3300/0 conn(0x7f87740c97b0 0x7f87740c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.229+0000 7f8773fff640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 0x7f87740c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.233+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258762491 0 0) 0x7f8774158c80 con 0x7f87740c8720 2024-03-16T20:00:16.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.233+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c003530 con 0x7f87740c8720 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025587733 0 0) 0x7f875c003530 con 0x7f87740c8720 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 msgr2=0x7f87740c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 0x7f87740c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 >> v2:172.21.15.59:3300/0 conn(0x7f87740c97b0 msgr2=0x7f87740c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 --2- 172.21.15.32:0/3205650112 >> v2:172.21.15.59:3300/0 conn(0x7f87740c97b0 0x7f87740c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877415bfc0 con 0x7f87740c8720 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f876c001a00 con 0x7f87740c8720 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 >> v1:172.21.15.32:6789/0 conn(0x7f87740c8720 legacy=0x7f877415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f875c003980 0x7f875c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f875c0047d0 0x7f875c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 0x7f875c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b0ad0 con 0x7f875c003980 2024-03-16T20:00:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8774157cb0 con 0x7f875c0047d0 2024-03-16T20:00:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.237+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875c00fdd0 con 0x7f8774151ef0 2024-03-16T20:00:16.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.241+0000 7f8773fff640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f875c003980 0x7f875c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f8773fff640 1 -- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 msgr2=0x7f875c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f8773fff640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8774151ef0 0x7f875c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f87737fe640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f875c0047d0 0x7f875c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f8773fff640 1 -- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f875c0047d0 msgr2=0x7f875c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:16.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f8773fff640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f875c0047d0 0x7f875c004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.245+0000 7f8773fff640 1 -- 172.21.15.32:0/3205650112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c003530 con 0x7f875c003980 2024-03-16T20:00:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.249+0000 7f8773fff640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f875c003980 0x7f875c003da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8764009ae0 tx=0x7f876401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b5744f16f9cbcb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640a55c0 con 0x7f875c003980 2024-03-16T20:00:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87640a5720 con 0x7f875c003980 2024-03-16T20:00:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f8772ffd640 1 -- 172.21.15.32:0/3205650112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640a5ad0 con 0x7f875c003980 2024-03-16T20:00:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 -- 172.21.15.32:0/3205650112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f875c003980 msgr2=0x7f875c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 --2- 172.21.15.32:0/3205650112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f875c003980 0x7f875c003da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8764009ae0 tx=0x7f876401cab0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f8778f90640 1 -- 172.21.15.32:0/3205650112 reap_dead start 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 -- 172.21.15.32:0/3205650112 shutdown_connections 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 -- 172.21.15.32:0/3205650112 >> 172.21.15.32:0/3205650112 conn(0x7f8774070f90 msgr2=0x7f87740713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 -- 172.21.15.32:0/3205650112 shutdown_connections 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.253+0000 7f877a9f3640 1 -- 172.21.15.32:0/3205650112 wait complete. 2024-03-16T20:00:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 Processor -- start 2024-03-16T20:00:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 -- start start 2024-03-16T20:00:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f875c003980 0x7f87740cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8774149db0 0x7f877406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877414c630 con 0x7f87740ccdc0 2024-03-16T20:00:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877414d460 con 0x7f8774149db0 2024-03-16T20:00:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f877a9f3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87740cca60 con 0x7f875c003980 2024-03-16T20:00:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f8778f90640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f87737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8774149db0 0x7f877406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f8778f90640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55364/0 (socket says 172.21.15.32:55364) 2024-03-16T20:00:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f8778f90640 1 -- 172.21.15.32:0/1406688731 learned_addr learned my addr 172.21.15.32:0/1406688731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.257+0000 7f8773fff640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f875c003980 0x7f87740cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.261+0000 7f8778f90640 1 -- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f875c003980 msgr2=0x7f87740cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.261+0000 7f8778f90640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f875c003980 0x7f87740cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.261+0000 7f8778f90640 1 -- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8774149db0 msgr2=0x7f877406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.261+0000 7f8778f90640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8774149db0 0x7f877406d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.261+0000 7f8778f90640 1 -- 172.21.15.32:0/1406688731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87740cd300 con 0x7f87740ccdc0 2024-03-16T20:00:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.265+0000 7f8778f90640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f87740cb3a0 tx=0x7f876c00ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876c00da80 con 0x7f87740ccdc0 2024-03-16T20:00:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876c00dc00 con 0x7f87740ccdc0 2024-03-16T20:00:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87740cb760 con 0x7f87740ccdc0 2024-03-16T20:00:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876c00a0d0 con 0x7f87740ccdc0 2024-03-16T20:00:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87740cbd20 con 0x7f87740ccdc0 2024-03-16T20:00:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.269+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87740b39e0 con 0x7f87740ccdc0 2024-03-16T20:00:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.305+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f876c00a230 con 0x7f87740ccdc0 2024-03-16T20:00:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.305+0000 7f8770ff9640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 0x7f8760069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.305+0000 7f8773fff640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 0x7f8760069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:16.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.309+0000 7f8773fff640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 0x7f8760069330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87640af000 tx=0x7f87640046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:16.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.613+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f876c0584c0 con 0x7f87740ccdc0 2024-03-16T20:00:16.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.613+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f876c054760 con 0x7f87740ccdc0 2024-03-16T20:00:16.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.857+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8774068000 con 0x7f87740ccdc0 2024-03-16T20:00:16.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f8770ff9640 1 -- 172.21.15.32:0/1406688731 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f876c01b070 con 0x7f87740ccdc0 2024-03-16T20:00:16.907 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:16.907 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:16.907 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 msgr2=0x7f8760069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 0x7f8760069330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87640af000 tx=0x7f87640046d0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 msgr2=0x7f8774149300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f87740cb3a0 tx=0x7f876c00ff00 comp rx=0 tx=0).stop 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 shutdown_connections 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8760066e70 0x7f8760069330 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87740ccdc0 0x7f8774149300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8774149db0 0x7f877406d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 --2- 172.21.15.32:0/1406688731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f875c003980 0x7f87740cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:16.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 >> 172.21.15.32:0/1406688731 conn(0x7f8774070f90 msgr2=0x7f87740c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:16.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.937+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 shutdown_connections 2024-03-16T20:00:16.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:16.941+0000 7f877a9f3640 1 -- 172.21.15.32:0/1406688731 wait complete. 2024-03-16T20:00:17.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:17.531 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:18.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:18.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 Processor -- start 2024-03-16T20:00:18.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 -- start start 2024-03-16T20:00:18.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c78150d70 0x7f3c78073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c780725e0 0x7f3c78070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c78072dd0 con 0x7f3c780708a0 2024-03-16T20:00:18.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c78070cb0 con 0x7f3c78150d70 2024-03-16T20:00:18.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c7e079640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7815d790 con 0x7f3c780725e0 2024-03-16T20:00:18.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c77fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3c780708a0 0x7f3c7815a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:55868/0 (socket says 172.21.15.32:55868) 2024-03-16T20:00:18.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.669+0000 7f3c77fff640 1 -- 172.21.15.32:0/1605036470 learned_addr learned my addr 172.21.15.32:0/1605036470 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:18.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.673+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/1605036470 >> v2:172.21.15.59:3300/0 conn(0x7f3c780725e0 0x7f3c78070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:18.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.681+0000 7f3c76ffd640 1 -- 172.21.15.32:0/1605036470 >> v2:172.21.15.32:3300/0 conn(0x7f3c78150d70 msgr2=0x7f3c78073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:18.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.681+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/1605036470 >> v2:172.21.15.32:3300/0 conn(0x7f3c78150d70 0x7f3c78073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:18.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.681+0000 7f3c76ffd640 1 -- 172.21.15.32:0/1605036470 >> v1:172.21.15.59:6789/0 conn(0x7f3c780708a0 legacy=0x7f3c7815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:18.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.681+0000 7f3c76ffd640 1 -- 172.21.15.32:0/1605036470 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7815da50 con 0x7f3c780725e0 2024-03-16T20:00:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.689+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/1605036470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780725e0 0x7f3c78070360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c7806ef10 tx=0x7f3c6c01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=886aec9325f9f2f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c767fc640 1 -- 172.21.15.32:0/1605036470 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c0ab600 con 0x7f3c780725e0 2024-03-16T20:00:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c767fc640 1 -- 172.21.15.32:0/1605036470 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c0a7050 con 0x7f3c780725e0 2024-03-16T20:00:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c767fc640 1 -- 172.21.15.32:0/1605036470 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c0ab980 con 0x7f3c780725e0 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- 172.21.15.32:0/1605036470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780725e0 msgr2=0x7f3c78070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 --2- 172.21.15.32:0/1605036470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780725e0 0x7f3c78070360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c7806ef10 tx=0x7f3c6c01c330 comp rx=0 tx=0).stop 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- 172.21.15.32:0/1605036470 shutdown_connections 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 --2- 172.21.15.32:0/1605036470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780725e0 0x7f3c78070360 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 --2- 172.21.15.32:0/1605036470 >> v2:172.21.15.32:3300/0 conn(0x7f3c78150d70 0x7f3c78073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- 172.21.15.32:0/1605036470 >> 172.21.15.32:0/1605036470 conn(0x7f3c7806c0d0 msgr2=0x7f3c7806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- 172.21.15.32:0/1605036470 shutdown_connections 2024-03-16T20:00:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- 172.21.15.32:0/1605036470 wait complete. 2024-03-16T20:00:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 Processor -- start 2024-03-16T20:00:18.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.701+0000 7f3c7e079640 1 -- start start 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3c780725e0 0x7f3c7815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c78150d70 0x7f3c7815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7814beb0 con 0x7f3c780708a0 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c781f6990 con 0x7f3c78150d70 2024-03-16T20:00:18.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c7e079640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c78148ec0 con 0x7f3c780725e0 2024-03-16T20:00:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c777fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c777fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55382/0 (socket says 172.21.15.32:55382) 2024-03-16T20:00:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c777fe640 1 -- 172.21.15.32:0/3158206009 learned_addr learned my addr 172.21.15.32:0/3158206009 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3c780725e0 0x7f3c7815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:18.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.705+0000 7f3c77fff640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c78150d70 0x7f3c7815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:18.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.709+0000 7f3c777fe640 1 -- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3c780725e0 msgr2=0x7f3c7815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:18.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.709+0000 7f3c777fe640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3c780725e0 0x7f3c7815c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:18.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.709+0000 7f3c777fe640 1 -- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c78150d70 msgr2=0x7f3c7815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:18.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.709+0000 7f3c777fe640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c78150d70 0x7f3c7815c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:18.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.709+0000 7f3c777fe640 1 -- 172.21.15.32:0/3158206009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c009420 con 0x7f3c780708a0 2024-03-16T20:00:18.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.713+0000 7f3c777fe640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800b550 tx=0x7f3c6800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.717+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68010870 con 0x7f3c780708a0 2024-03-16T20:00:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.717+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680109f0 con 0x7f3c780708a0 2024-03-16T20:00:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.717+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68010da0 con 0x7f3c780708a0 2024-03-16T20:00:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.717+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7815d560 con 0x7f3c780708a0 2024-03-16T20:00:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.717+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c78149180 con 0x7f3c780708a0 2024-03-16T20:00:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:18.721+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c78068000 con 0x7f3c780708a0 2024-03-16T20:00:18.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.005+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3c68023480 con 0x7f3c780708a0 2024-03-16T20:00:18.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.005+0000 7f3c5ffff640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 0x7f3c64069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:18.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.005+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 0x7f3c64069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:18.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.005+0000 7f3c76ffd640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 0x7f3c64069350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ad000 tx=0x7f3c6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:19.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.033+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3c68055a50 con 0x7f3c780708a0 2024-03-16T20:00:19.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.033+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3c68051cf0 con 0x7f3c780708a0 2024-03-16T20:00:19.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.233+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c780722b0 con 0x7f3c780708a0 2024-03-16T20:00:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.253+0000 7f3c5ffff640 1 -- 172.21.15.32:0/3158206009 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f3c6801e090 con 0x7f3c780708a0 2024-03-16T20:00:19.226 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:19.226 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:19.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 msgr2=0x7f3c64069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 0x7f3c64069350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ad000 tx=0x7f3c6c008040 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 msgr2=0x7f3c78149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800b550 tx=0x7f3c6800ba10 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 shutdown_connections 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3c64066e90 0x7f3c64069350 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c78150d70 0x7f3c7815c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3c780725e0 0x7f3c7815c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 --2- 172.21.15.32:0/3158206009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780708a0 0x7f3c78149750 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 >> 172.21.15.32:0/3158206009 conn(0x7f3c7806c0d0 msgr2=0x7f3c78143c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 shutdown_connections 2024-03-16T20:00:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:19.257+0000 7f3c7e079640 1 -- 172.21.15.32:0/3158206009 wait complete. 2024-03-16T20:00:19.855 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:19.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:20.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:20.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 Processor -- start 2024-03-16T20:00:20.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 -- start start 2024-03-16T20:00:20.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7a28150d70 0x7f7a28073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a2815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a28072dd0 con 0x7f7a280725e0 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2815d790 con 0x7f7a28150d70 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2cde3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815da50 con 0x7f7a2815a420 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a2659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7a28150d70 0x7f7a28073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a25d9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a280725e0 0x7f7a28070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34184/0 (socket says 172.21.15.32:34184) 2024-03-16T20:00:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a25d9b640 1 -- 172.21.15.32:0/1966319985 learned_addr learned my addr 172.21.15.32:0/1966319985 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:20.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.993+0000 7f7a26d9d640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a2815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.997+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432599631 0 0) 0x7f7a28072dd0 con 0x7f7a280725e0 2024-03-16T20:00:20.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:20.997+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a10003530 con 0x7f7a280725e0 2024-03-16T20:00:20.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2897285522 0 0) 0x7f7a10003530 con 0x7f7a280725e0 2024-03-16T20:00:20.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 msgr2=0x7f7a2815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a2815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 >> v2:172.21.15.59:3300/0 conn(0x7f7a28150d70 msgr2=0x7f7a28073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 --2- 172.21.15.32:0/1966319985 >> v2:172.21.15.59:3300/0 conn(0x7f7a28150d70 0x7f7a28073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2815dd30 con 0x7f7a280725e0 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f7a1c001680 con 0x7f7a280725e0 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 >> v1:172.21.15.32:6789/0 conn(0x7f7a280725e0 legacy=0x7f7a28070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a10003980 0x7f7a10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a100047d0 0x7f7a10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815d790 con 0x7f7a100047d0 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815da50 con 0x7f7a10003980 2024-03-16T20:00:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.001+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1000ba70 con 0x7f7a2815a420 2024-03-16T20:00:20.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.005+0000 7f7a25d9b640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a100047d0 0x7f7a10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.005+0000 7f7a26d9d640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a2659c640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a10003980 0x7f7a10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a25d9b640 1 -- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 msgr2=0x7f7a10005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:20.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a25d9b640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a2815a420 0x7f7a10005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a25d9b640 1 -- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a10003980 msgr2=0x7f7a10003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:20.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a25d9b640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a10003980 0x7f7a10003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.009+0000 7f7a25d9b640 1 -- 172.21.15.32:0/1966319985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10003530 con 0x7f7a100047d0 2024-03-16T20:00:20.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.013+0000 7f7a25d9b640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a100047d0 0x7f7a10004c50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c001060 tx=0x7f7a1c01cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f73efa18b239d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:20.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0058e0 con 0x7f7a100047d0 2024-03-16T20:00:20.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c005a40 con 0x7f7a100047d0 2024-03-16T20:00:20.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2559a640 1 -- 172.21.15.32:0/1966319985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c005df0 con 0x7f7a100047d0 2024-03-16T20:00:20.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 -- 172.21.15.32:0/1966319985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a100047d0 msgr2=0x7f7a10004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/1966319985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a100047d0 0x7f7a10004c50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c001060 tx=0x7f7a1c01cb10 comp rx=0 tx=0).stop 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a26d9d640 1 -- 172.21.15.32:0/1966319985 reap_dead start 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 -- 172.21.15.32:0/1966319985 shutdown_connections 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 -- 172.21.15.32:0/1966319985 >> 172.21.15.32:0/1966319985 conn(0x7f7a2806c0d0 msgr2=0x7f7a2806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 -- 172.21.15.32:0/1966319985 shutdown_connections 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.017+0000 7f7a2cde3640 1 -- 172.21.15.32:0/1966319985 wait complete. 2024-03-16T20:00:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 Processor -- start 2024-03-16T20:00:20.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 -- start start 2024-03-16T20:00:20.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a100047d0 0x7f7a280c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a280c62b0 0x7f7a28141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28142bc0 con 0x7f7a2813f480 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28145550 con 0x7f7a100047d0 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2cde3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a281410b0 con 0x7f7a280c62b0 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a26d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a26d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37750/0 (socket says 172.21.15.32:37750) 2024-03-16T20:00:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a26d9d640 1 -- 172.21.15.32:0/581594350 learned_addr learned my addr 172.21.15.32:0/581594350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a2659c640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a100047d0 0x7f7a280c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.021+0000 7f7a25d9b640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a280c62b0 0x7f7a28141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.025+0000 7f7a26d9d640 1 -- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a280c62b0 msgr2=0x7f7a28141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.025+0000 7f7a26d9d640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a280c62b0 0x7f7a28141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.025+0000 7f7a26d9d640 1 -- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a100047d0 msgr2=0x7f7a280c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.025+0000 7f7a26d9d640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a100047d0 0x7f7a280c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.025+0000 7f7a26d9d640 1 -- 172.21.15.32:0/581594350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2815d560 con 0x7f7a2813f480 2024-03-16T20:00:20.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.029+0000 7f7a26d9d640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a2806e2d0 tx=0x7f7a200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:21.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2000e070 con 0x7f7a2813f480 2024-03-16T20:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a20011ea0 con 0x7f7a2813f480 2024-03-16T20:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200039a0 con 0x7f7a2813f480 2024-03-16T20:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2814ee70 con 0x7f7a2813f480 2024-03-16T20:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a28142600 con 0x7f7a2813f480 2024-03-16T20:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.033+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a28068000 con 0x7f7a2813f480 2024-03-16T20:00:21.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.073+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a20003b00 con 0x7f7a2813f480 2024-03-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.073+0000 7f7a16ffd640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 0x7f7a08069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.073+0000 7f7a2659c640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 0x7f7a08069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.073+0000 7f7a2659c640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 0x7f7a08069350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009ae0 tx=0x7f7a18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.469+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7a20056a10 con 0x7f7a2813f480 2024-03-16T20:00:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.469+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7a20052cb0 con 0x7f7a2813f480 2024-03-16T20:00:21.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.697+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a281411f0 con 0x7f7a2813f480 2024-03-16T20:00:21.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.721+0000 7f7a16ffd640 1 -- 172.21.15.32:0/581594350 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f7a20056370 con 0x7f7a2813f480 2024-03-16T20:00:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:21.694 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 msgr2=0x7f7a08069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 0x7f7a08069350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009ae0 tx=0x7f7a18008040 comp rx=0 tx=0).stop 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 msgr2=0x7f7a2813f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a2806e2d0 tx=0x7f7a200115a0 comp rx=0 tx=0).stop 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 shutdown_connections 2024-03-16T20:00:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a08066e90 0x7f7a08069350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a2813f480 0x7f7a2813f900 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a280c62b0 0x7f7a28141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 --2- 172.21.15.32:0/581594350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a100047d0 0x7f7a280c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 >> 172.21.15.32:0/581594350 conn(0x7f7a2806c0d0 msgr2=0x7f7a2814f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 shutdown_connections 2024-03-16T20:00:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:21.725+0000 7f7a2cde3640 1 -- 172.21.15.32:0/581594350 wait complete. 2024-03-16T20:00:22.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:22.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:23.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 Processor -- start 2024-03-16T20:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 -- start start 2024-03-16T20:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a280c6420 0x7f3a280c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a28070760 con 0x7f3a28158390 2024-03-16T20:00:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a280731c0 con 0x7f3a280c6040 2024-03-16T20:00:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a2dd2f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a280734a0 con 0x7f3a280c6420 2024-03-16T20:00:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a26ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a280c6420 0x7f3a280c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a26ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a280c6420 0x7f3a280c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37752/0 (socket says 172.21.15.32:37752) 2024-03-16T20:00:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.457+0000 7f3a26ffd640 1 -- 172.21.15.32:0/2949444302 learned_addr learned my addr 172.21.15.32:0/2949444302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.461+0000 7f3a26ffd640 1 -- 172.21.15.32:0/2949444302 >> v1:172.21.15.59:6789/0 conn(0x7f3a280c6040 legacy=0x7f3a28157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.461+0000 7f3a26ffd640 1 -- 172.21.15.32:0/2949444302 >> v1:172.21.15.32:6789/0 conn(0x7f3a28158390 legacy=0x7f3a281516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.461+0000 7f3a26ffd640 1 -- 172.21.15.32:0/2949444302 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2806fe30 con 0x7f3a280c6420 2024-03-16T20:00:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.465+0000 7f3a26ffd640 1 --2- 172.21.15.32:0/2949444302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6420 0x7f3a280c8800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009870 tx=0x7f3a1c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469040eeb6ee1f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a267fc640 1 -- 172.21.15.32:0/2949444302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c0aa650 con 0x7f3a280c6420 2024-03-16T20:00:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a267fc640 1 -- 172.21.15.32:0/2949444302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c0aa7b0 con 0x7f3a280c6420 2024-03-16T20:00:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a267fc640 1 -- 172.21.15.32:0/2949444302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c0aab40 con 0x7f3a280c6420 2024-03-16T20:00:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/2949444302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6420 msgr2=0x7f3a280c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/2949444302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6420 0x7f3a280c8800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009870 tx=0x7f3a1c01c4f0 comp rx=0 tx=0).stop 2024-03-16T20:00:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/2949444302 shutdown_connections 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/2949444302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6420 0x7f3a280c8800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/2949444302 >> 172.21.15.32:0/2949444302 conn(0x7f3a2806c0d0 msgr2=0x7f3a2806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/2949444302 shutdown_connections 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.469+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/2949444302 wait complete. 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 Processor -- start 2024-03-16T20:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 -- start start 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a280c6420 0x7f3a28145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a28158390 0x7f3a281464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a281f3670 con 0x7f3a280c6040 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a281f27e0 con 0x7f3a28158390 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a2dd2f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a281f29f0 con 0x7f3a280c6420 2024-03-16T20:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a277fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37766/0 (socket says 172.21.15.32:37766) 2024-03-16T20:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a277fe640 1 -- 172.21.15.32:0/329269233 learned_addr learned my addr 172.21.15.32:0/329269233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a27fff640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a28158390 0x7f3a281464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.473+0000 7f3a26ffd640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a280c6420 0x7f3a28145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.477+0000 7f3a277fe640 1 -- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a280c6420 msgr2=0x7f3a28145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.477+0000 7f3a277fe640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a280c6420 0x7f3a28145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.477+0000 7f3a277fe640 1 -- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a28158390 msgr2=0x7f3a281464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.477+0000 7f3a277fe640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a28158390 0x7f3a281464e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.477+0000 7f3a277fe640 1 -- 172.21.15.32:0/329269233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0094d0 con 0x7f3a280c6040 2024-03-16T20:00:23.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.481+0000 7f3a277fe640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c370 tx=0x7f3a1800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:23.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a180052d0 con 0x7f3a280c6040 2024-03-16T20:00:23.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a18005430 con 0x7f3a280c6040 2024-03-16T20:00:23.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a281f2cd0 con 0x7f3a280c6040 2024-03-16T20:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18007260 con 0x7f3a280c6040 2024-03-16T20:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a281f3290 con 0x7f3a280c6040 2024-03-16T20:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.485+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28068000 con 0x7f3a280c6040 2024-03-16T20:00:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.521+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3a180073c0 con 0x7f3a280c6040 2024-03-16T20:00:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.521+0000 7f3a0ffff640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 0x7f3a14069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.521+0000 7f3a26ffd640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 0x7f3a14069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.521+0000 7f3a26ffd640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 0x7f3a14069300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0ae000 tx=0x7f3a1c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.817+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3a18007670 con 0x7f3a280c6040 2024-03-16T20:00:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:23.817+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3a18051100 con 0x7f3a280c6040 2024-03-16T20:00:23.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.017+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a280c7d40 con 0x7f3a280c6040 2024-03-16T20:00:24.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.037+0000 7f3a0ffff640 1 -- 172.21.15.32:0/329269233 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f3a180547c0 con 0x7f3a280c6040 2024-03-16T20:00:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:24.008 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:24.008 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 msgr2=0x7f3a14069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 0x7f3a14069300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0ae000 tx=0x7f3a1c0b5040 comp rx=0 tx=0).stop 2024-03-16T20:00:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 msgr2=0x7f3a28156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c370 tx=0x7f3a1800c830 comp rx=0 tx=0).stop 2024-03-16T20:00:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 shutdown_connections 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3a14066e40 0x7f3a14069300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a28158390 0x7f3a281464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3a280c6420 0x7f3a28145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 --2- 172.21.15.32:0/329269233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a280c6040 0x7f3a28156d60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 >> 172.21.15.32:0/329269233 conn(0x7f3a2806c0d0 msgr2=0x7f3a280c90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 shutdown_connections 2024-03-16T20:00:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:24.041+0000 7f3a2dd2f640 1 -- 172.21.15.32:0/329269233 wait complete. 2024-03-16T20:00:24.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:24.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:25.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 Processor -- start 2024-03-16T20:00:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 -- start start 2024-03-16T20:00:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f72ac153fa0 0x7f72ac070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72ac0724a0 0x7f72ac0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0c8590 con 0x7f72ac0c7470 2024-03-16T20:00:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ac0c3f00 con 0x7f72ac0724a0 2024-03-16T20:00:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.769+0000 7f72b38ee640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ac070650 con 0x7f72ac153fa0 2024-03-16T20:00:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72ac0724a0 0x7f72ac0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72ac0724a0 0x7f72ac0728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37778/0 (socket says 172.21.15.32:37778) 2024-03-16T20:00:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 -- 172.21.15.32:0/1805925129 learned_addr learned my addr 172.21.15.32:0/1805925129 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b168a640 1 --2- 172.21.15.32:0/1805925129 >> v2:172.21.15.59:3300/0 conn(0x7f72ac153fa0 0x7f72ac070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 -- 172.21.15.32:0/1805925129 >> v2:172.21.15.59:3300/0 conn(0x7f72ac153fa0 msgr2=0x7f72ac070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:25.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 --2- 172.21.15.32:0/1805925129 >> v2:172.21.15.59:3300/0 conn(0x7f72ac153fa0 0x7f72ac070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 -- 172.21.15.32:0/1805925129 >> v1:172.21.15.59:6789/0 conn(0x7f72ac0c7470 legacy=0x7f72ac15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:25.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.773+0000 7f72b1e8b640 1 -- 172.21.15.32:0/1805925129 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac073590 con 0x7f72ac0724a0 2024-03-16T20:00:25.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.777+0000 7f72b1e8b640 1 --2- 172.21.15.32:0/1805925129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac0728f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f72ac070230 tx=0x7f72a401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0305f662ef3afe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.781+0000 7f72a3fff640 1 -- 172.21.15.32:0/1805925129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a40af400 con 0x7f72ac0724a0 2024-03-16T20:00:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.781+0000 7f72a3fff640 1 -- 172.21.15.32:0/1805925129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a40ab050 con 0x7f72ac0724a0 2024-03-16T20:00:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.781+0000 7f72a3fff640 1 -- 172.21.15.32:0/1805925129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a40af780 con 0x7f72ac0724a0 2024-03-16T20:00:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.781+0000 7f72b38ee640 1 -- 172.21.15.32:0/1805925129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 msgr2=0x7f72ac0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.781+0000 7f72b38ee640 1 --2- 172.21.15.32:0/1805925129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac0728f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f72ac070230 tx=0x7f72a401f1a0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- 172.21.15.32:0/1805925129 shutdown_connections 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 --2- 172.21.15.32:0/1805925129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac0728f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 --2- 172.21.15.32:0/1805925129 >> v2:172.21.15.59:3300/0 conn(0x7f72ac153fa0 0x7f72ac070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- 172.21.15.32:0/1805925129 >> 172.21.15.32:0/1805925129 conn(0x7f72ac06c0d0 msgr2=0x7f72ac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- 172.21.15.32:0/1805925129 shutdown_connections 2024-03-16T20:00:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- 172.21.15.32:0/1805925129 wait complete. 2024-03-16T20:00:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 Processor -- start 2024-03-16T20:00:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- start start 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ac0c7470 0x7f72ac150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72ac153fa0 0x7f72ac15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac150ca0 con 0x7f72ac0724a0 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac150f80 con 0x7f72ac0c7470 2024-03-16T20:00:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b38ee640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac151260 con 0x7f72ac153fa0 2024-03-16T20:00:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b168a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b1e8b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72ac153fa0 0x7f72ac15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b168a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37784/0 (socket says 172.21.15.32:37784) 2024-03-16T20:00:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b168a640 1 -- 172.21.15.32:0/3010984091 learned_addr learned my addr 172.21.15.32:0/3010984091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.785+0000 7f72b0e89640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ac0c7470 0x7f72ac150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.793+0000 7f72b168a640 1 -- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72ac153fa0 msgr2=0x7f72ac15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.793+0000 7f72b168a640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72ac153fa0 0x7f72ac15d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.793+0000 7f72b168a640 1 -- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ac0c7470 msgr2=0x7f72ac150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.793+0000 7f72b168a640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ac0c7470 0x7f72ac150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.793+0000 7f72b168a640 1 -- 172.21.15.32:0/3010984091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009b30 con 0x7f72ac0724a0 2024-03-16T20:00:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.797+0000 7f72b168a640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d5f0 tx=0x7f72a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80125f0 con 0x7f72ac0724a0 2024-03-16T20:00:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8012750 con 0x7f72ac0724a0 2024-03-16T20:00:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8012ae0 con 0x7f72ac0724a0 2024-03-16T20:00:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac15cde0 con 0x7f72ac0724a0 2024-03-16T20:00:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac15d3a0 con 0x7f72ac0724a0 2024-03-16T20:00:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:25.801+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac0c78a0 con 0x7f72ac0724a0 2024-03-16T20:00:26.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.105+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f72a8012c40 con 0x7f72ac0724a0 2024-03-16T20:00:26.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.105+0000 7f72a1ffb640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 0x7f7290069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:26.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.105+0000 7f72b0e89640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 0x7f7290069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:26.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.105+0000 7f72b0e89640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 0x7f7290069300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72ac153350 tx=0x7f729c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:26.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.133+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f72a8056380 con 0x7f72ac0724a0 2024-03-16T20:00:26.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.133+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f72a8052620 con 0x7f72ac0724a0 2024-03-16T20:00:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.341+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72ac0c78a0 con 0x7f72ac0724a0 2024-03-16T20:00:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.361+0000 7f72a1ffb640 1 -- 172.21.15.32:0/3010984091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f72ac0c78a0 con 0x7f72ac0724a0 2024-03-16T20:00:26.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:26.334 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 msgr2=0x7f7290069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 0x7f7290069300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72ac153350 tx=0x7f729c007e40 comp rx=0 tx=0).stop 2024-03-16T20:00:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 msgr2=0x7f72ac14f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d5f0 tx=0x7f72a800dab0 comp rx=0 tx=0).stop 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 shutdown_connections 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e40 0x7f7290069300 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72ac153fa0 0x7f72ac15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ac0c7470 0x7f72ac150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 --2- 172.21.15.32:0/3010984091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72ac0724a0 0x7f72ac14f1b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 >> 172.21.15.32:0/3010984091 conn(0x7f72ac06c0d0 msgr2=0x7f72ac06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 shutdown_connections 2024-03-16T20:00:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:26.365+0000 7f72b38ee640 1 -- 172.21.15.32:0/3010984091 wait complete. 2024-03-16T20:00:26.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:26.956 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:27.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 Processor -- start 2024-03-16T20:00:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 -- start start 2024-03-16T20:00:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdb08157550 0x7fdb08162460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb08150fa0 con 0x7fdb08073120 2024-03-16T20:00:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb08071b00 con 0x7fdb08150bc0 2024-03-16T20:00:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0ec3a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb080c7c30 con 0x7fdb08157550 2024-03-16T20:00:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb07fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb08073120 0x7fdb080c69e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34194/0 (socket says 172.21.15.32:34194) 2024-03-16T20:00:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb07fff640 1 -- 172.21.15.32:0/1628520222 learned_addr learned my addr 172.21.15.32:0/1628520222 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.093+0000 7fdb0d1d7640 1 --2- 172.21.15.32:0/1628520222 >> v2:172.21.15.59:3300/0 conn(0x7fdb08157550 0x7fdb08162460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.097+0000 7fdb077fe640 1 -- 172.21.15.32:0/1628520222 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488698523 0 0) 0x7fdb08150fa0 con 0x7fdb08073120 2024-03-16T20:00:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.097+0000 7fdb077fe640 1 -- 172.21.15.32:0/1628520222 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf8003530 con 0x7fdb08073120 2024-03-16T20:00:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.105+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/1628520222 >> v1:172.21.15.59:6789/0 conn(0x7fdb08150bc0 legacy=0x7fdb080c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.105+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/1628520222 >> v1:172.21.15.32:6789/0 conn(0x7fdb08073120 legacy=0x7fdb080c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.105+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/1628520222 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08071da0 con 0x7fdb08157550 2024-03-16T20:00:28.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.113+0000 7fdb0d1d7640 1 --2- 172.21.15.32:0/1628520222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08157550 0x7fdb08162460 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009d90 tx=0x7fdb0001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cdab84f638aa6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.121+0000 7fdb077fe640 1 -- 172.21.15.32:0/1628520222 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000af600 con 0x7fdb08157550 2024-03-16T20:00:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.121+0000 7fdb077fe640 1 -- 172.21.15.32:0/1628520222 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb000ab050 con 0x7fdb08157550 2024-03-16T20:00:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.121+0000 7fdb077fe640 1 -- 172.21.15.32:0/1628520222 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000af980 con 0x7fdb08157550 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/1628520222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08157550 msgr2=0x7fdb08162460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/1628520222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08157550 0x7fdb08162460 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009d90 tx=0x7fdb0001f150 comp rx=0 tx=0).stop 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/1628520222 shutdown_connections 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/1628520222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08157550 0x7fdb08162460 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/1628520222 >> 172.21.15.32:0/1628520222 conn(0x7fdb080cbfe0 msgr2=0x7fdb080cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/1628520222 shutdown_connections 2024-03-16T20:00:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/1628520222 wait complete. 2024-03-16T20:00:28.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 Processor -- start 2024-03-16T20:00:28.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- start start 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08073120 0x7fdb081e3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb08150bc0 0x7fdb080c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081efe80 con 0x7fdb08157550 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081e2eb0 con 0x7fdb08073120 2024-03-16T20:00:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0ec3a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081f0e50 con 0x7fdb08150bc0 2024-03-16T20:00:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0d1d7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0d1d7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37800/0 (socket says 172.21.15.32:37800) 2024-03-16T20:00:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/3324710790 learned_addr learned my addr 172.21.15.32:0/3324710790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.125+0000 7fdb07fff640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb08150bc0 0x7fdb080c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.129+0000 7fdb0c9d6640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08073120 0x7fdb081e3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.133+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb08150bc0 msgr2=0x7fdb080c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.133+0000 7fdb0d1d7640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb08150bc0 0x7fdb080c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.133+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08073120 msgr2=0x7fdb081e3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.133+0000 7fdb0d1d7640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08073120 0x7fdb081e3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.133+0000 7fdb0d1d7640 1 -- 172.21.15.32:0/3324710790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00009a70 con 0x7fdb08157550 2024-03-16T20:00:28.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.137+0000 7fdb0d1d7640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdb0001cd80 tx=0x7fdb0001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000afbf0 con 0x7fdb08157550 2024-03-16T20:00:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb081e3ab0 con 0x7fdb08157550 2024-03-16T20:00:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb081f0370 con 0x7fdb08157550 2024-03-16T20:00:28.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb08068000 con 0x7fdb08157550 2024-03-16T20:00:28.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb000afd50 con 0x7fdb08157550 2024-03-16T20:00:28.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.141+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000cd680 con 0x7fdb08157550 2024-03-16T20:00:28.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.169+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdb000cd7e0 con 0x7fdb08157550 2024-03-16T20:00:28.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.169+0000 7fdb057fa640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 0x7fdaec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:28.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.169+0000 7fdb0c9d6640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 0x7fdaec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:28.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.169+0000 7fdb0c9d6640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 0x7fdaec069300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdafc007fd0 tx=0x7fdafc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.465+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdb000f4130 con 0x7fdb08157550 2024-03-16T20:00:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.465+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdb000f03d0 con 0x7fdb08157550 2024-03-16T20:00:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.677+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0814e510 con 0x7fdb08157550 2024-03-16T20:00:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb057fa640 1 -- 172.21.15.32:0/3324710790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fdb000b4020 con 0x7fdb08157550 2024-03-16T20:00:28.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:28.672 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:28.673 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 msgr2=0x7fdaec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 0x7fdaec069300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdafc007fd0 tx=0x7fdafc009040 comp rx=0 tx=0).stop 2024-03-16T20:00:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 msgr2=0x7fdb081f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdb0001cd80 tx=0x7fdb0001bef0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 shutdown_connections 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdaec066e40 0x7fdaec069300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb08157550 0x7fdb081f33f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb08150bc0 0x7fdb080c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 --2- 172.21.15.32:0/3324710790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb08073120 0x7fdb081e3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 >> 172.21.15.32:0/3324710790 conn(0x7fdb080cbfe0 msgr2=0x7fdb080cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 shutdown_connections 2024-03-16T20:00:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:28.701+0000 7fdb0ec3a640 1 -- 172.21.15.32:0/3324710790 wait complete. 2024-03-16T20:00:29.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:29.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:30.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 Processor -- start 2024-03-16T20:00:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 -- start start 2024-03-16T20:00:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7320150c40 0x7f7320073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7320151020 con 0x7f73200725e0 2024-03-16T20:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7320072da0 con 0x7f732015a420 2024-03-16T20:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.429+0000 7f73270c8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732015a000 con 0x7f7320150c40 2024-03-16T20:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.433+0000 7f7324e64640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7320150c40 0x7f7320073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.433+0000 7f731ffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73200725e0 0x7f7320070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35552/0 (socket says 172.21.15.32:35552) 2024-03-16T20:00:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.433+0000 7f731ffff640 1 -- 172.21.15.32:0/1389022171 learned_addr learned my addr 172.21.15.32:0/1389022171 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.437+0000 7f7324e64640 1 -- 172.21.15.32:0/1389022171 >> v1:172.21.15.59:6789/0 conn(0x7f732015a420 legacy=0x7f732015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.437+0000 7f7324e64640 1 -- 172.21.15.32:0/1389022171 >> v1:172.21.15.32:6789/0 conn(0x7f73200725e0 legacy=0x7f7320070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.437+0000 7f7324e64640 1 -- 172.21.15.32:0/1389022171 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732015e310 con 0x7f7320150c40 2024-03-16T20:00:30.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.441+0000 7f7324e64640 1 --2- 172.21.15.32:0/1389022171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7320150c40 0x7f7320073070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f73140097c0 tx=0x7f731401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce1e6d3d26e97ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f731f7fe640 1 -- 172.21.15.32:0/1389022171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140a6040 con 0x7f7320150c40 2024-03-16T20:00:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f731f7fe640 1 -- 172.21.15.32:0/1389022171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73140aa600 con 0x7f7320150c40 2024-03-16T20:00:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f731f7fe640 1 -- 172.21.15.32:0/1389022171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140aa990 con 0x7f7320150c40 2024-03-16T20:00:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 -- 172.21.15.32:0/1389022171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7320150c40 msgr2=0x7f7320073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 --2- 172.21.15.32:0/1389022171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7320150c40 0x7f7320073070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f73140097c0 tx=0x7f731401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 -- 172.21.15.32:0/1389022171 shutdown_connections 2024-03-16T20:00:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 --2- 172.21.15.32:0/1389022171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7320150c40 0x7f7320073070 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 -- 172.21.15.32:0/1389022171 >> 172.21.15.32:0/1389022171 conn(0x7f732006c0d0 msgr2=0x7f732006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 -- 172.21.15.32:0/1389022171 shutdown_connections 2024-03-16T20:00:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.445+0000 7f73270c8640 1 -- 172.21.15.32:0/1389022171 wait complete. 2024-03-16T20:00:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 Processor -- start 2024-03-16T20:00:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 -- start start 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73200725e0 0x7f732015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7320150c40 0x7f7320147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732013d4e0 con 0x7f732015a420 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7320068710 con 0x7f73200725e0 2024-03-16T20:00:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f73270c8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732014efe0 con 0x7f7320150c40 2024-03-16T20:00:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f7325665640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f7325665640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33186/0 (socket says 172.21.15.32:33186) 2024-03-16T20:00:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f7325665640 1 -- 172.21.15.32:0/3839021249 learned_addr learned my addr 172.21.15.32:0/3839021249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f7324e64640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73200725e0 0x7f732015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.449+0000 7f731ffff640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7320150c40 0x7f7320147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.453+0000 7f7325665640 1 -- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7320150c40 msgr2=0x7f7320147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.453+0000 7f7325665640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7320150c40 0x7f7320147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.453+0000 7f7325665640 1 -- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73200725e0 msgr2=0x7f732015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.453+0000 7f7325665640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73200725e0 0x7f732015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.453+0000 7f7325665640 1 -- 172.21.15.32:0/3839021249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009420 con 0x7f732015a420 2024-03-16T20:00:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.457+0000 7f7325665640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f731800c770 tx=0x7f731800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:30.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.461+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318003b80 con 0x7f732015a420 2024-03-16T20:00:30.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.461+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7318003ce0 con 0x7f732015a420 2024-03-16T20:00:30.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.461+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732015d560 con 0x7f732015a420 2024-03-16T20:00:30.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.461+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73201f68c0 con 0x7f732015a420 2024-03-16T20:00:30.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.461+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731800aa90 con 0x7f732015a420 2024-03-16T20:00:30.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.465+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73000052a0 con 0x7f732015a420 2024-03-16T20:00:30.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.497+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f731800abf0 con 0x7f732015a420 2024-03-16T20:00:30.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.497+0000 7f731d7fa640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 0x7f7304069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:30.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.497+0000 7f7324e64640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 0x7f7304069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:30.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.497+0000 7f7324e64640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 0x7f7304069350 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73140ae000 tx=0x7f73140b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.765+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7318055e10 con 0x7f732015a420 2024-03-16T20:00:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.765+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f73180520b0 con 0x7f732015a420 2024-03-16T20:00:30.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.977+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7300005900 con 0x7f732015a420 2024-03-16T20:00:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.989+0000 7f731d7fa640 1 -- 172.21.15.32:0/3839021249 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7f7300005900 con 0x7f732015a420 2024-03-16T20:00:30.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:30.962 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 msgr2=0x7f7304069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 0x7f7304069350 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73140ae000 tx=0x7f73140b5040 comp rx=0 tx=0).stop 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 msgr2=0x7f732013bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f731800c770 tx=0x7f731800cc30 comp rx=0 tx=0).stop 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 shutdown_connections 2024-03-16T20:00:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7304066e90 0x7f7304069350 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f732015a420 0x7f732013bf00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7320150c40 0x7f7320147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 --2- 172.21.15.32:0/3839021249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73200725e0 0x7f732015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 >> 172.21.15.32:0/3839021249 conn(0x7f732006c0d0 msgr2=0x7f73200c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 shutdown_connections 2024-03-16T20:00:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:30.993+0000 7f73270c8640 1 -- 172.21.15.32:0/3839021249 wait complete. 2024-03-16T20:00:31.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:31.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:32.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 Processor -- start 2024-03-16T20:00:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 -- start start 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc194150d70 0x7fc194073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc1940725e0 0x7fc194070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1940708a0 0x7fc19415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc194072dd0 con 0x7fc1940708a0 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19415d790 con 0x7fc194150d70 2024-03-16T20:00:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19c0be640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19415da50 con 0x7fc1940725e0 2024-03-16T20:00:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc199e5a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc194150d70 0x7fc194073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.741+0000 7fc19a65b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1940708a0 0x7fc19415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1940708a0 0x7fc19415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33202/0 (socket says 172.21.15.32:33202) 2024-03-16T20:00:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 -- 172.21.15.32:0/4017088904 learned_addr learned my addr 172.21.15.32:0/4017088904 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 -- 172.21.15.32:0/4017088904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc1940725e0 msgr2=0x7fc194070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 --2- 172.21.15.32:0/4017088904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc1940725e0 0x7fc194070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 -- 172.21.15.32:0/4017088904 >> v2:172.21.15.59:3300/0 conn(0x7fc194150d70 msgr2=0x7fc194073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 --2- 172.21.15.32:0/4017088904 >> v2:172.21.15.59:3300/0 conn(0x7fc194150d70 0x7fc194073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.745+0000 7fc19a65b640 1 -- 172.21.15.32:0/4017088904 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19415dd30 con 0x7fc1940708a0 2024-03-16T20:00:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.749+0000 7fc19a65b640 1 --2- 172.21.15.32:0/4017088904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940708a0 0x7fc19415a420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc19406c080 tx=0x7fc18c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34e6abb154be3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc198e58640 1 -- 172.21.15.32:0/4017088904 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0a57e0 con 0x7fc1940708a0 2024-03-16T20:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc198e58640 1 -- 172.21.15.32:0/4017088904 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c0a5940 con 0x7fc1940708a0 2024-03-16T20:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc198e58640 1 -- 172.21.15.32:0/4017088904 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0a5cd0 con 0x7fc1940708a0 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 -- 172.21.15.32:0/4017088904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940708a0 msgr2=0x7fc19415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 --2- 172.21.15.32:0/4017088904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940708a0 0x7fc19415a420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc19406c080 tx=0x7fc18c01c960 comp rx=0 tx=0).stop 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 -- 172.21.15.32:0/4017088904 shutdown_connections 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 --2- 172.21.15.32:0/4017088904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940708a0 0x7fc19415a420 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 --2- 172.21.15.32:0/4017088904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc1940725e0 0x7fc194070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 --2- 172.21.15.32:0/4017088904 >> v2:172.21.15.59:3300/0 conn(0x7fc194150d70 0x7fc194073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 -- 172.21.15.32:0/4017088904 >> 172.21.15.32:0/4017088904 conn(0x7fc19406c0d0 msgr2=0x7fc19406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 -- 172.21.15.32:0/4017088904 shutdown_connections 2024-03-16T20:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.757+0000 7fc19c0be640 1 -- 172.21.15.32:0/4017088904 wait complete. 2024-03-16T20:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 Processor -- start 2024-03-16T20:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 -- start start 2024-03-16T20:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940725e0 0x7fc194152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc194150d70 0x7fc1941464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941516a0 con 0x7fc1940725e0 2024-03-16T20:00:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc194153cf0 con 0x7fc1940708a0 2024-03-16T20:00:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19c0be640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f6a40 con 0x7fc194150d70 2024-03-16T20:00:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc199e5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc19a65b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc194150d70 0x7fc1941464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc199e5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:52036/0 (socket says 172.21.15.32:52036) 2024-03-16T20:00:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.761+0000 7fc199e5a640 1 -- 172.21.15.32:0/482876607 learned_addr learned my addr 172.21.15.32:0/482876607 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.769+0000 7fc199e5a640 1 -- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc194150d70 msgr2=0x7fc1941464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.769+0000 7fc199e5a640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc194150d70 0x7fc1941464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.769+0000 7fc199e5a640 1 -- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940725e0 msgr2=0x7fc194152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.769+0000 7fc199e5a640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940725e0 0x7fc194152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:32.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.769+0000 7fc199e5a640 1 -- 172.21.15.32:0/482876607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c009b30 con 0x7fc1940708a0 2024-03-16T20:00:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.773+0000 7fc199e5a640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e870 tx=0x7fc19000ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190009510 con 0x7fc1940708a0 2024-03-16T20:00:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19000eec0 con 0x7fc1940708a0 2024-03-16T20:00:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19415d560 con 0x7fc1940708a0 2024-03-16T20:00:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19000cb40 con 0x7fc1940708a0 2024-03-16T20:00:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941f6ca0 con 0x7fc1940708a0 2024-03-16T20:00:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:32.781+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc194068000 con 0x7fc1940708a0 2024-03-16T20:00:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.117+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc19000cd40 con 0x7fc1940708a0 2024-03-16T20:00:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.117+0000 7fc18a7fc640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 0x7fc178069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.117+0000 7fc199659640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 0x7fc178069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.121+0000 7fc199659640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 0x7fc178069380 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc19414dfc0 tx=0x7fc184006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:33.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.121+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc190055dd0 con 0x7fc1940708a0 2024-03-16T20:00:33.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.137+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc190052070 con 0x7fc1940708a0 2024-03-16T20:00:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.405+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1940722b0 con 0x7fc1940708a0 2024-03-16T20:00:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.441+0000 7fc18a7fc640 1 -- 172.21.15.32:0/482876607 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6037 (secure 0 0 0) 0x7fc190017090 con 0x7fc1940708a0 2024-03-16T20:00:33.415 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:33.415 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T19:58:45.556744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:40.485778+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:00:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 msgr2=0x7fc178069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 0x7fc178069380 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc19414dfc0 tx=0x7fc184006ca0 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 msgr2=0x7fc19414f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e870 tx=0x7fc19000ec70 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 shutdown_connections 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc178066ec0 0x7fc178069380 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc194150d70 0x7fc1941464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1940725e0 0x7fc194152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 --2- 172.21.15.32:0/482876607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1940708a0 0x7fc19414f170 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 >> 172.21.15.32:0/482876607 conn(0x7fc19406c0d0 msgr2=0x7fc1940cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 shutdown_connections 2024-03-16T20:00:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:33.445+0000 7fc19c0be640 1 -- 172.21.15.32:0/482876607 wait complete. 2024-03-16T20:00:34.039 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T19:58:45.556744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:40.485778+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:00:34.039 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:35.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:35.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 Processor -- start 2024-03-16T20:00:35.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 -- start start 2024-03-16T20:00:35.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a40c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:35.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a4070760 con 0x7fb7a40c6040 2024-03-16T20:00:35.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a40731c0 con 0x7fb7a40c6420 2024-03-16T20:00:35.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7aafb3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40734a0 con 0x7fb7a4158390 2024-03-16T20:00:35.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7a8d4f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb7a40c6040 0x7fb7a4157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35562/0 (socket says 172.21.15.32:35562) 2024-03-16T20:00:35.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7a3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb7a40c6420 0x7fb7a41516d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:46472/0 (socket says 172.21.15.32:46472) 2024-03-16T20:00:35.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7a8d4f640 1 -- 172.21.15.32:0/4176111805 learned_addr learned my addr 172.21.15.32:0/4176111805 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:35.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.181+0000 7fb7a9550640 1 --2- 172.21.15.32:0/4176111805 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a40c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:35.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.185+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398224016 0 0) 0x7fb7a4070760 con 0x7fb7a40c6040 2024-03-16T20:00:35.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.185+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb794003530 con 0x7fb7a40c6040 2024-03-16T20:00:35.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.185+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963777085 0 0) 0x7fb7a40731c0 con 0x7fb7a40c6420 2024-03-16T20:00:35.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.185+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a4070760 con 0x7fb7a40c6420 2024-03-16T20:00:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.189+0000 7fb7a9550640 1 -- 172.21.15.32:0/4176111805 >> v1:172.21.15.59:6789/0 conn(0x7fb7a40c6420 legacy=0x7fb7a41516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.189+0000 7fb7a9550640 1 -- 172.21.15.32:0/4176111805 >> v1:172.21.15.32:6789/0 conn(0x7fb7a40c6040 legacy=0x7fb7a4157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.189+0000 7fb7a9550640 1 -- 172.21.15.32:0/4176111805 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a406fe30 con 0x7fb7a4158390 2024-03-16T20:00:35.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.193+0000 7fb7a9550640 1 --2- 172.21.15.32:0/4176111805 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a40c8800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009dc0 tx=0x7fb79c01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a9cc9946c7bac90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a5a20 con 0x7fb7a4158390 2024-03-16T20:00:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c0a5b80 con 0x7fb7a4158390 2024-03-16T20:00:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7a37fe640 1 -- 172.21.15.32:0/4176111805 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0ad440 con 0x7fb7a4158390 2024-03-16T20:00:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 -- 172.21.15.32:0/4176111805 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 msgr2=0x7fb7a40c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/4176111805 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a40c8800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009dc0 tx=0x7fb79c01c960 comp rx=0 tx=0).stop 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 -- 172.21.15.32:0/4176111805 shutdown_connections 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/4176111805 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a40c8800 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 -- 172.21.15.32:0/4176111805 >> 172.21.15.32:0/4176111805 conn(0x7fb7a406c0d0 msgr2=0x7fb7a406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 -- 172.21.15.32:0/4176111805 shutdown_connections 2024-03-16T20:00:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.205+0000 7fb7aafb3640 1 -- 172.21.15.32:0/4176111805 wait complete. 2024-03-16T20:00:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 Processor -- start 2024-03-16T20:00:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 -- start start 2024-03-16T20:00:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40c6420 0x7fb7a40c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a41f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a413ef50 con 0x7fb7a40c6040 2024-03-16T20:00:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a41f6f60 con 0x7fb7a40c6420 2024-03-16T20:00:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7aafb3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4150290 con 0x7fb7a4158390 2024-03-16T20:00:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7a8d4f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7a8d4f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33216/0 (socket says 172.21.15.32:33216) 2024-03-16T20:00:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7a8d4f640 1 -- 172.21.15.32:0/3495777461 learned_addr learned my addr 172.21.15.32:0/3495777461 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40c6420 0x7fb7a40c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:35.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.209+0000 7fb7a9550640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a41f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.213+0000 7fb7a8d4f640 1 -- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 msgr2=0x7fb7a41f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.213+0000 7fb7a8d4f640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a41f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.213+0000 7fb7a8d4f640 1 -- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40c6420 msgr2=0x7fb7a40c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.213+0000 7fb7a8d4f640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40c6420 0x7fb7a40c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.213+0000 7fb7a8d4f640 1 -- 172.21.15.32:0/3495777461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009a70 con 0x7fb7a40c6040 2024-03-16T20:00:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.217+0000 7fb7a8d4f640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c530 tx=0x7fb79800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798005cf0 con 0x7fb7a40c6040 2024-03-16T20:00:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798005e50 con 0x7fb7a40c6040 2024-03-16T20:00:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798007330 con 0x7fb7a40c6040 2024-03-16T20:00:35.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a4146490 con 0x7fb7a40c6040 2024-03-16T20:00:35.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a414eb40 con 0x7fb7a40c6040 2024-03-16T20:00:35.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.225+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4068000 con 0x7fb7a40c6040 2024-03-16T20:00:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.261+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb798007490 con 0x7fb7a40c6040 2024-03-16T20:00:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.261+0000 7fb7a17fa640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 0x7fb788069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.261+0000 7fb7a3fff640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 0x7fb788069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.261+0000 7fb7a3fff640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 0x7fb788069300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4149c70 tx=0x7fb790007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:35.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.265+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb798007740 con 0x7fb7a40c6040 2024-03-16T20:00:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.549+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb798051330 con 0x7fb7a40c6040 2024-03-16T20:00:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.865+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a406fa50 con 0x7fb7a40c6040 2024-03-16T20:00:35.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.889+0000 7fb7a17fa640 1 -- 172.21.15.32:0/3495777461 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb798017030 con 0x7fb7a40c6040 2024-03-16T20:00:35.861 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:35.862 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:35.862 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:35.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 msgr2=0x7fb788069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 0x7fb788069300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4149c70 tx=0x7fb790007d40 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 msgr2=0x7fb7a4145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c530 tx=0x7fb79800c9f0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 shutdown_connections 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb788066e40 0x7fb788069300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb7a4158390 0x7fb7a41f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40c6420 0x7fb7a40c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 --2- 172.21.15.32:0/3495777461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40c6040 0x7fb7a4145f50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 >> 172.21.15.32:0/3495777461 conn(0x7fb7a406c0d0 msgr2=0x7fb7a4155b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 shutdown_connections 2024-03-16T20:00:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:35.893+0000 7fb7aafb3640 1 -- 172.21.15.32:0/3495777461 wait complete. 2024-03-16T20:00:36.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:36.484 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:37.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 Processor -- start 2024-03-16T20:00:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 -- start start 2024-03-16T20:00:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd7c151ef0 0x7efd7c072d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd7c0c7800 0x7efd7c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c15b740 con 0x7efd7c15b2d0 2024-03-16T20:00:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0c70d0 con 0x7efd7c151ef0 2024-03-16T20:00:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.621+0000 7efd809d9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c072a00 con 0x7efd7c0c7800 2024-03-16T20:00:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.625+0000 7efd7a59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd7c151ef0 0x7efd7c072d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.625+0000 7efd7ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd7c0c7800 0x7efd7c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.625+0000 7efd7a59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd7c151ef0 0x7efd7c072d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33232/0 (socket says 172.21.15.32:33232) 2024-03-16T20:00:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.625+0000 7efd7a59c640 1 -- 172.21.15.32:0/1020525327 learned_addr learned my addr 172.21.15.32:0/1020525327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:37.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.629+0000 7efd7a59c640 1 -- 172.21.15.32:0/1020525327 >> v2:172.21.15.59:3300/0 conn(0x7efd7c0c7800 msgr2=0x7efd7c0c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:37.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.629+0000 7efd7a59c640 1 --2- 172.21.15.32:0/1020525327 >> v2:172.21.15.59:3300/0 conn(0x7efd7c0c7800 0x7efd7c0c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:37.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.629+0000 7efd7a59c640 1 -- 172.21.15.32:0/1020525327 >> v1:172.21.15.59:6789/0 conn(0x7efd7c15b2d0 legacy=0x7efd7c0722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:37.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.629+0000 7efd7a59c640 1 -- 172.21.15.32:0/1020525327 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0c7c50 con 0x7efd7c151ef0 2024-03-16T20:00:37.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.633+0000 7efd7a59c640 1 --2- 172.21.15.32:0/1020525327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c151ef0 0x7efd7c072d50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0c8df0 tx=0x7efd6c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7422aa535317d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd7959a640 1 -- 172.21.15.32:0/1020525327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0a8a10 con 0x7efd7c151ef0 2024-03-16T20:00:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd7959a640 1 -- 172.21.15.32:0/1020525327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c0a8b70 con 0x7efd7c151ef0 2024-03-16T20:00:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd7959a640 1 -- 172.21.15.32:0/1020525327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0b0440 con 0x7efd7c151ef0 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 -- 172.21.15.32:0/1020525327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c151ef0 msgr2=0x7efd7c072d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 --2- 172.21.15.32:0/1020525327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c151ef0 0x7efd7c072d50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0c8df0 tx=0x7efd6c01ed10 comp rx=0 tx=0).stop 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 -- 172.21.15.32:0/1020525327 shutdown_connections 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 --2- 172.21.15.32:0/1020525327 >> v2:172.21.15.59:3300/0 conn(0x7efd7c0c7800 0x7efd7c0c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 --2- 172.21.15.32:0/1020525327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c151ef0 0x7efd7c072d50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 -- 172.21.15.32:0/1020525327 >> 172.21.15.32:0/1020525327 conn(0x7efd7c06e6c0 msgr2=0x7efd7c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 -- 172.21.15.32:0/1020525327 shutdown_connections 2024-03-16T20:00:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.637+0000 7efd809d9640 1 -- 172.21.15.32:0/1020525327 wait complete. 2024-03-16T20:00:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 Processor -- start 2024-03-16T20:00:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 -- start start 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c151ef0 0x7efd7c1401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efd7c15b2d0 0x7efd7c153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cca90 con 0x7efd7c0c7800 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cd500 con 0x7efd7c151ef0 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd809d9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c1f02b0 con 0x7efd7c15b2d0 2024-03-16T20:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd7a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd7a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33238/0 (socket says 172.21.15.32:33238) 2024-03-16T20:00:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd7a59c640 1 -- 172.21.15.32:0/3966978369 learned_addr learned my addr 172.21.15.32:0/3966978369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd79d9b640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c151ef0 0x7efd7c1401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.641+0000 7efd7ad9d640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efd7c15b2d0 0x7efd7c153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.645+0000 7efd7a59c640 1 -- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efd7c15b2d0 msgr2=0x7efd7c153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.645+0000 7efd7a59c640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efd7c15b2d0 0x7efd7c153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.645+0000 7efd7a59c640 1 -- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c151ef0 msgr2=0x7efd7c1401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.645+0000 7efd7a59c640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c151ef0 0x7efd7c1401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.645+0000 7efd7a59c640 1 -- 172.21.15.32:0/3966978369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0094d0 con 0x7efd7c0c7800 2024-03-16T20:00:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.649+0000 7efd7a59c640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd6c01b920 tx=0x7efd6c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:37.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.653+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0ba600 con 0x7efd7c0c7800 2024-03-16T20:00:37.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.653+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c0ba760 con 0x7efd7c0c7800 2024-03-16T20:00:37.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.653+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c1f07e0 con 0x7efd7c0c7800 2024-03-16T20:00:37.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.653+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0bab10 con 0x7efd7c0c7800 2024-03-16T20:00:37.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.657+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c15a930 con 0x7efd7c0c7800 2024-03-16T20:00:37.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.657+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c068000 con 0x7efd7c0c7800 2024-03-16T20:00:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.689+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efd6c0bac70 con 0x7efd7c0c7800 2024-03-16T20:00:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.689+0000 7efd6affd640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 0x7efd5c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.689+0000 7efd79d9b640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 0x7efd5c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.689+0000 7efd79d9b640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 0x7efd5c069330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd7c153550 tx=0x7efd70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.693+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efd6c0f46b0 con 0x7efd7c0c7800 2024-03-16T20:00:37.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:37.969+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efd6c0f0950 con 0x7efd7c0c7800 2024-03-16T20:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.169+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd7c14cfc0 con 0x7efd7c0c7800 2024-03-16T20:00:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.193+0000 7efd6affd640 1 -- 172.21.15.32:0/3966978369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7efd6c0b5070 con 0x7efd7c0c7800 2024-03-16T20:00:38.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:38.167 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 msgr2=0x7efd5c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 0x7efd5c069330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd7c153550 tx=0x7efd70009040 comp rx=0 tx=0).stop 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 msgr2=0x7efd7c15a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd6c01b920 tx=0x7efd6c01bab0 comp rx=0 tx=0).stop 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 shutdown_connections 2024-03-16T20:00:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efd5c066e70 0x7efd5c069330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efd7c15b2d0 0x7efd7c153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c151ef0 0x7efd7c1401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 --2- 172.21.15.32:0/3966978369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0c7800 0x7efd7c15a3f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 >> 172.21.15.32:0/3966978369 conn(0x7efd7c06e6c0 msgr2=0x7efd7c06bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 shutdown_connections 2024-03-16T20:00:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:38.197+0000 7efd809d9640 1 -- 172.21.15.32:0/3966978369 wait complete. 2024-03-16T20:00:38.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:38.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:39.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 Processor -- start 2024-03-16T20:00:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 -- start start 2024-03-16T20:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff478151000 0x7ff4780c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4780c8ec0 con 0x7ff4780c8980 2024-03-16T20:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff478072950 con 0x7ff4780c36b0 2024-03-16T20:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.925+0000 7ff4805ca640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff478072f10 con 0x7ff478151000 2024-03-16T20:00:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.929+0000 7ff47eb67640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff4780c8980 0x7ff4781615d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54574/0 (socket says 172.21.15.32:54574) 2024-03-16T20:00:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.929+0000 7ff47eb67640 1 -- 172.21.15.32:0/580217890 learned_addr learned my addr 172.21.15.32:0/580217890 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.929+0000 7ff47e366640 1 --2- 172.21.15.32:0/580217890 >> v2:172.21.15.59:3300/0 conn(0x7ff478151000 0x7ff4780c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.929+0000 7ff47d364640 1 -- 172.21.15.32:0/580217890 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706520040 0 0) 0x7ff4780c8ec0 con 0x7ff4780c8980 2024-03-16T20:00:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.929+0000 7ff47d364640 1 -- 172.21.15.32:0/580217890 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff464003530 con 0x7ff4780c8980 2024-03-16T20:00:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.933+0000 7ff47e366640 1 -- 172.21.15.32:0/580217890 >> v1:172.21.15.59:6789/0 conn(0x7ff4780c36b0 legacy=0x7ff4780c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.933+0000 7ff47e366640 1 -- 172.21.15.32:0/580217890 >> v1:172.21.15.32:6789/0 conn(0x7ff4780c8980 legacy=0x7ff4781615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.933+0000 7ff47e366640 1 -- 172.21.15.32:0/580217890 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478072c30 con 0x7ff478151000 2024-03-16T20:00:39.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.937+0000 7ff47e366640 1 --2- 172.21.15.32:0/580217890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff478151000 0x7ff4780c8420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097c0 tx=0x7ff47401c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b65a75bb00734d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.945+0000 7ff47d364640 1 -- 172.21.15.32:0/580217890 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474004230 con 0x7ff478151000 2024-03-16T20:00:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.945+0000 7ff47d364640 1 -- 172.21.15.32:0/580217890 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4740a7050 con 0x7ff478151000 2024-03-16T20:00:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.945+0000 7ff47d364640 1 -- 172.21.15.32:0/580217890 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474003620 con 0x7ff478151000 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- 172.21.15.32:0/580217890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff478151000 msgr2=0x7ff4780c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 --2- 172.21.15.32:0/580217890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff478151000 0x7ff4780c8420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097c0 tx=0x7ff47401c460 comp rx=0 tx=0).stop 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- 172.21.15.32:0/580217890 shutdown_connections 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 --2- 172.21.15.32:0/580217890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff478151000 0x7ff4780c8420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- 172.21.15.32:0/580217890 >> 172.21.15.32:0/580217890 conn(0x7ff47806c0d0 msgr2=0x7ff47806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- 172.21.15.32:0/580217890 shutdown_connections 2024-03-16T20:00:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- 172.21.15.32:0/580217890 wait complete. 2024-03-16T20:00:39.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 Processor -- start 2024-03-16T20:00:39.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.949+0000 7ff4805ca640 1 -- start start 2024-03-16T20:00:39.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff4780c36b0 0x7ff47806b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:39.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780c8980 0x7ff47814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:39.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:39.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff478160ee0 con 0x7ff478151000 2024-03-16T20:00:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47806bda0 con 0x7ff4780c8980 2024-03-16T20:00:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff4805ca640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47814ebb0 con 0x7ff4780c36b0 2024-03-16T20:00:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff47eb67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff47eb67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54608/0 (socket says 172.21.15.32:54608) 2024-03-16T20:00:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff47eb67640 1 -- 172.21.15.32:0/2588924046 learned_addr learned my addr 172.21.15.32:0/2588924046 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff47e366640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff4780c36b0 0x7ff47806b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.953+0000 7ff47db65640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780c8980 0x7ff47814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.957+0000 7ff47eb67640 1 -- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff4780c36b0 msgr2=0x7ff47806b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:39.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.957+0000 7ff47eb67640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff4780c36b0 0x7ff47806b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:39.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.957+0000 7ff47eb67640 1 -- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780c8980 msgr2=0x7ff47814f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:39.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.957+0000 7ff47eb67640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780c8980 0x7ff47814f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:39.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.957+0000 7ff47eb67640 1 -- 172.21.15.32:0/2588924046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474009420 con 0x7ff478151000 2024-03-16T20:00:39.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.961+0000 7ff47eb67640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff470001590 tx=0x7ff47000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:39.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.965+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47000ac00 con 0x7ff478151000 2024-03-16T20:00:39.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.965+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47000ad80 con 0x7ff478151000 2024-03-16T20:00:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.965+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff478161db0 con 0x7ff478151000 2024-03-16T20:00:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.965+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470012640 con 0x7ff478151000 2024-03-16T20:00:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.965+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47814ee50 con 0x7ff478151000 2024-03-16T20:00:39.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.969+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47814f060 con 0x7ff478151000 2024-03-16T20:00:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.989+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff47001c440 con 0x7ff478151000 2024-03-16T20:00:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.989+0000 7ff46effd640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 0x7ff45c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.989+0000 7ff47e366640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 0x7ff45c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:39.989+0000 7ff47e366640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 0x7ff45c069350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff4740ad000 tx=0x7ff4740a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:40.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.285+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff470055c10 con 0x7ff478151000 2024-03-16T20:00:40.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.285+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff470051eb0 con 0x7ff478151000 2024-03-16T20:00:40.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.525+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff478068000 con 0x7ff478151000 2024-03-16T20:00:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.545+0000 7ff46effd640 1 -- 172.21.15.32:0/2588924046 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7ff470055570 con 0x7ff478151000 2024-03-16T20:00:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:40.519 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:40.519 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:40.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 msgr2=0x7ff45c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:40.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 0x7ff45c069350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff4740ad000 tx=0x7ff4740a6000 comp rx=0 tx=0).stop 2024-03-16T20:00:40.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 msgr2=0x7ff478142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff470001590 tx=0x7ff47000cf90 comp rx=0 tx=0).stop 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 shutdown_connections 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff45c066e90 0x7ff45c069350 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff478151000 0x7ff478142dc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780c8980 0x7ff47814f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 --2- 172.21.15.32:0/2588924046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff4780c36b0 0x7ff47806b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 >> 172.21.15.32:0/2588924046 conn(0x7ff47806c0d0 msgr2=0x7ff4780c5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 shutdown_connections 2024-03-16T20:00:40.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:40.549+0000 7ff4805ca640 1 -- 172.21.15.32:0/2588924046 wait complete. 2024-03-16T20:00:41.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:41.142 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:41.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:00:41.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:00:42.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:42.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.281+0000 7f58ea24c640 1 Processor -- start 2024-03-16T20:00:42.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.281+0000 7f58ea24c640 1 -- start start 2024-03-16T20:00:42.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58ea24c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58e4071130 0x7f58e40c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58ea24c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e4071510 0x7f58e40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58ea24c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40c2940 con 0x7f58e40c8fe0 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58ea24c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e40b2de0 con 0x7f58e4071130 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58ea24c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40ca1f0 con 0x7f58e4071510 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58e3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f58e40c8fe0 0x7f58e4151b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49450/0 (socket says 172.21.15.32:49450) 2024-03-16T20:00:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58e3fff640 1 -- 172.21.15.32:0/4019854245 learned_addr learned my addr 172.21.15.32:0/4019854245 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:42.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58e37fe640 1 --2- 172.21.15.32:0/4019854245 >> v2:172.21.15.32:3300/0 conn(0x7f58e4071130 0x7f58e40c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.285+0000 7f58e2ffd640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e4071510 0x7f58e40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.289+0000 7f58e37fe640 1 -- 172.21.15.32:0/4019854245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e4071510 msgr2=0x7f58e40c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.289+0000 7f58e37fe640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e4071510 0x7f58e40c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.289+0000 7f58e37fe640 1 -- 172.21.15.32:0/4019854245 >> v1:172.21.15.59:6789/0 conn(0x7f58e40c8fe0 legacy=0x7f58e4151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.289+0000 7f58e37fe640 1 -- 172.21.15.32:0/4019854245 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40c2b90 con 0x7f58e4071130 2024-03-16T20:00:42.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.293+0000 7f58e37fe640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e40c9b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f58d40097c0 tx=0x7f58d401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c8b930f8cb5b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.293+0000 7f58e27fc640 1 -- 172.21.15.32:0/4019854245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d40a8990 con 0x7f58e4071130 2024-03-16T20:00:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58e27fc640 1 -- 172.21.15.32:0/4019854245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d40a8af0 con 0x7f58e4071130 2024-03-16T20:00:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58e27fc640 1 -- 172.21.15.32:0/4019854245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d40a8e80 con 0x7f58e4071130 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 -- 172.21.15.32:0/4019854245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 msgr2=0x7f58e40c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e40c9b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f58d40097c0 tx=0x7f58d401ec90 comp rx=0 tx=0).stop 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 -- 172.21.15.32:0/4019854245 shutdown_connections 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e4071510 0x7f58e40c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 --2- 172.21.15.32:0/4019854245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e40c9b90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 -- 172.21.15.32:0/4019854245 >> 172.21.15.32:0/4019854245 conn(0x7f58e406b190 msgr2=0x7f58e406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 -- 172.21.15.32:0/4019854245 shutdown_connections 2024-03-16T20:00:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.297+0000 7f58ea24c640 1 -- 172.21.15.32:0/4019854245 wait complete. 2024-03-16T20:00:42.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 Processor -- start 2024-03-16T20:00:42.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 -- start start 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e4071510 0x7f58e41f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e40c8fe0 0x7f58e4152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e41526a0 con 0x7f58e4071130 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e41ef310 con 0x7f58e4071510 2024-03-16T20:00:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58ea24c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e41ef5f0 con 0x7f58e40c8fe0 2024-03-16T20:00:42.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54622/0 (socket says 172.21.15.32:54622) 2024-03-16T20:00:42.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58e37fe640 1 -- 172.21.15.32:0/1891099621 learned_addr learned my addr 172.21.15.32:0/1891099621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:42.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.301+0000 7f58e2ffd640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e4071510 0x7f58e41f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e3fff640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e40c8fe0 0x7f58e4152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e37fe640 1 -- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e40c8fe0 msgr2=0x7f58e4152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e37fe640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e40c8fe0 0x7f58e4152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e37fe640 1 -- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e4071510 msgr2=0x7f58e41f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e37fe640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e4071510 0x7f58e41f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.305+0000 7f58e37fe640 1 -- 172.21.15.32:0/1891099621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4009420 con 0x7f58e4071130 2024-03-16T20:00:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.309+0000 7f58e37fe640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f58d40b3040 tx=0x7f58d401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.313+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d40b9630 con 0x7f58e4071130 2024-03-16T20:00:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.313+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d40b97b0 con 0x7f58e4071130 2024-03-16T20:00:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.313+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e41ef8d0 con 0x7f58e4071130 2024-03-16T20:00:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.313+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d40b9b60 con 0x7f58e4071130 2024-03-16T20:00:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.313+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e41efe90 con 0x7f58e4071130 2024-03-16T20:00:42.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.317+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e4068000 con 0x7f58e4071130 2024-03-16T20:00:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.337+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58d40b7040 con 0x7f58e4071130 2024-03-16T20:00:42.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.337+0000 7f58cbfff640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 0x7f58d0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:42.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.337+0000 7f58e2ffd640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 0x7f58d0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:42.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.337+0000 7f58e2ffd640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 0x7f58d0069350 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f58e415c2c0 tx=0x7f58d8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:42.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.641+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f58d40f40e0 con 0x7f58e4071130 2024-03-16T20:00:42.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.641+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f58d40f0380 con 0x7f58e4071130 2024-03-16T20:00:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:42.912+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e414b2a0 con 0x7f58e4071130 2024-03-16T20:00:42.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58cbfff640 1 -- 172.21.15.32:0/1891099621 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f58d40b5030 con 0x7f58e4071130 2024-03-16T20:00:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:42.984 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 msgr2=0x7f58d0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 0x7f58d0069350 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f58e415c2c0 tx=0x7f58d8009190 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 msgr2=0x7f58e415ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f58d40b3040 tx=0x7f58d401ba30 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 shutdown_connections 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e90 0x7f58d0069350 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58e40c8fe0 0x7f58e4152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e4071510 0x7f58e41f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 --2- 172.21.15.32:0/1891099621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58e4071130 0x7f58e415ba20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 >> 172.21.15.32:0/1891099621 conn(0x7f58e406b190 msgr2=0x7f58e414f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 shutdown_connections 2024-03-16T20:00:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:43.012+0000 7f58ea24c640 1 -- 172.21.15.32:0/1891099621 wait complete. 2024-03-16T20:00:43.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:43.613 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:44.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:44.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.760+0000 7f6fcb23c640 1 Processor -- start 2024-03-16T20:00:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.760+0000 7f6fcb23c640 1 -- start start 2024-03-16T20:00:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fcb23c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6fc40725e0 0x7f6fc4073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fcb23c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc4072dd0 con 0x7f6fc4150d70 2024-03-16T20:00:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fcb23c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc415d790 con 0x7f6fc415a420 2024-03-16T20:00:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fcb23c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc415da50 con 0x7f6fc40725e0 2024-03-16T20:00:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fc8fd8640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6fc4150d70 0x7f6fc4070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54586/0 (socket says 172.21.15.32:54586) 2024-03-16T20:00:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fc8fd8640 1 -- 172.21.15.32:0/4064293624 learned_addr learned my addr 172.21.15.32:0/4064293624 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.764+0000 7f6fc3fff640 1 --2- 172.21.15.32:0/4064293624 >> v2:172.21.15.59:3300/0 conn(0x7f6fc40725e0 0x7f6fc4073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.768+0000 7f6fc37fe640 1 -- 172.21.15.32:0/4064293624 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033856421 0 0) 0x7f6fc4072dd0 con 0x7f6fc4150d70 2024-03-16T20:00:44.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.768+0000 7f6fc37fe640 1 -- 172.21.15.32:0/4064293624 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003530 con 0x7f6fc4150d70 2024-03-16T20:00:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.768+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4064293624 >> v1:172.21.15.59:6789/0 conn(0x7f6fc415a420 legacy=0x7f6fc415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.768+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4064293624 >> v1:172.21.15.32:6789/0 conn(0x7f6fc4150d70 legacy=0x7f6fc4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.768+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4064293624 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc415dd30 con 0x7f6fc40725e0 2024-03-16T20:00:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.772+0000 7f6fc3fff640 1 --2- 172.21.15.32:0/4064293624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc4073070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00097c0 tx=0x7f6fb001c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc8e1264747e3146 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:44.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fc37fe640 1 -- 172.21.15.32:0/4064293624 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00ac640 con 0x7f6fc40725e0 2024-03-16T20:00:44.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fc37fe640 1 -- 172.21.15.32:0/4064293624 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00ac7a0 con 0x7f6fc40725e0 2024-03-16T20:00:44.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fc37fe640 1 -- 172.21.15.32:0/4064293624 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00acb30 con 0x7f6fc40725e0 2024-03-16T20:00:44.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4064293624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 msgr2=0x7f6fc4073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4064293624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc4073070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00097c0 tx=0x7f6fb001c310 comp rx=0 tx=0).stop 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4064293624 shutdown_connections 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4064293624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc4073070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4064293624 >> 172.21.15.32:0/4064293624 conn(0x7f6fc406c0d0 msgr2=0x7f6fc406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4064293624 shutdown_connections 2024-03-16T20:00:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4064293624 wait complete. 2024-03-16T20:00:44.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 Processor -- start 2024-03-16T20:00:44.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- start start 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc40c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc415a420 0x7f6fc4142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc40c6a80 con 0x7f6fc4150d70 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc415ca60 con 0x7f6fc40725e0 2024-03-16T20:00:44.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.784+0000 7f6fcb23c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41479f0 con 0x7f6fc415a420 2024-03-16T20:00:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.788+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.788+0000 7f6fc97d9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc415a420 0x7f6fc4142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.788+0000 7f6fc8fd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc40c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.788+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54634/0 (socket says 172.21.15.32:54634) 2024-03-16T20:00:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.788+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4131027108 learned_addr learned my addr 172.21.15.32:0/4131027108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.792+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc415a420 msgr2=0x7f6fc4142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.792+0000 7f6fc3fff640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc415a420 0x7f6fc4142860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.792+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 msgr2=0x7f6fc40c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.792+0000 7f6fc3fff640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc40c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.792+0000 7f6fc3fff640 1 -- 172.21.15.32:0/4131027108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0009420 con 0x7f6fc4150d70 2024-03-16T20:00:44.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.796+0000 7f6fc3fff640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0018f20 tx=0x7f6fb00190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00b5610 con 0x7f6fc4150d70 2024-03-16T20:00:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00b5770 con 0x7f6fc4150d70 2024-03-16T20:00:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00b5b00 con 0x7f6fc4150d70 2024-03-16T20:00:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc415d560 con 0x7f6fc4150d70 2024-03-16T20:00:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc415cf90 con 0x7f6fc4150d70 2024-03-16T20:00:44.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.800+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc4068000 con 0x7f6fc4150d70 2024-03-16T20:00:44.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.840+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6fb00b3040 con 0x7f6fc4150d70 2024-03-16T20:00:44.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.840+0000 7f6fc17fa640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 0x7f6fa8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:44.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.840+0000 7f6fc8fd8640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 0x7f6fa8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:44.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:44.840+0000 7f6fc8fd8640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 0x7f6fa8069350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8007fd0 tx=0x7f6fb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:45.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.144+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6fb00efc80 con 0x7f6fc4150d70 2024-03-16T20:00:45.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.144+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6fb00ebf20 con 0x7f6fc4150d70 2024-03-16T20:00:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.400+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fc40c8180 con 0x7f6fc4150d70 2024-03-16T20:00:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.432+0000 7f6fc17fa640 1 -- 172.21.15.32:0/4131027108 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6fb00c3090 con 0x7f6fc4150d70 2024-03-16T20:00:45.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:45.408 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 msgr2=0x7f6fa8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 0x7f6fa8069350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8007fd0 tx=0x7f6fb8009040 comp rx=0 tx=0).stop 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 msgr2=0x7f6fc4142320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0018f20 tx=0x7f6fb00190b0 comp rx=0 tx=0).stop 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 shutdown_connections 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6fa8066e90 0x7f6fa8069350 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6fc415a420 0x7f6fc4142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fc4150d70 0x7f6fc4142320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 --2- 172.21.15.32:0/4131027108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc40725e0 0x7f6fc40c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 >> 172.21.15.32:0/4131027108 conn(0x7f6fc406c0d0 msgr2=0x7f6fc414a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 shutdown_connections 2024-03-16T20:00:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:45.436+0000 7f6fcb23c640 1 -- 172.21.15.32:0/4131027108 wait complete. 2024-03-16T20:00:46.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:46.031 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:47.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:47.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 Processor -- start 2024-03-16T20:00:47.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 -- start start 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2bd00c73f0 0x7f2bd0153e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd0154390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd015a390 con 0x7f2bd006de40 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0071b70 con 0x7f2bd00c73f0 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd7224640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd015af30 con 0x7f2bd006e220 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd4fc0640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2bd006de40 0x7f2bd015a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54592/0 (socket says 172.21.15.32:54592) 2024-03-16T20:00:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd4fc0640 1 -- 172.21.15.32:0/4012577028 learned_addr learned my addr 172.21.15.32:0/4012577028 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:47.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bd57c1640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd0154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.168+0000 7f2bcffff640 1 --2- 172.21.15.32:0/4012577028 >> v2:172.21.15.59:3300/0 conn(0x7f2bd00c73f0 0x7f2bd0153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.172+0000 7f2bcf7fe640 1 -- 172.21.15.32:0/4012577028 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910830266 0 0) 0x7f2bd015a390 con 0x7f2bd006de40 2024-03-16T20:00:47.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.172+0000 7f2bcf7fe640 1 -- 172.21.15.32:0/4012577028 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc0003530 con 0x7f2bd006de40 2024-03-16T20:00:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.176+0000 7f2bcffff640 1 -- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 msgr2=0x7f2bd0154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.176+0000 7f2bcffff640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd0154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.176+0000 7f2bcffff640 1 -- 172.21.15.32:0/4012577028 >> v1:172.21.15.32:6789/0 conn(0x7f2bd006de40 legacy=0x7f2bd015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.176+0000 7f2bcffff640 1 -- 172.21.15.32:0/4012577028 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0072bd0 con 0x7f2bd00c73f0 2024-03-16T20:00:47.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.180+0000 7f2bcffff640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00c73f0 0x7f2bd0153e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097c0 tx=0x7f2bbc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1772594a4af93d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.188+0000 7f2bcf7fe640 1 -- 172.21.15.32:0/4012577028 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0a8990 con 0x7f2bd00c73f0 2024-03-16T20:00:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.188+0000 7f2bcf7fe640 1 -- 172.21.15.32:0/4012577028 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc0a8af0 con 0x7f2bd00c73f0 2024-03-16T20:00:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.188+0000 7f2bcf7fe640 1 -- 172.21.15.32:0/4012577028 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0a8e80 con 0x7f2bd00c73f0 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00c73f0 msgr2=0x7f2bd0153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00c73f0 0x7f2bd0153e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097c0 tx=0x7f2bbc01ec90 comp rx=0 tx=0).stop 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- 172.21.15.32:0/4012577028 shutdown_connections 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd0154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- 172.21.15.32:0/4012577028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00c73f0 0x7f2bd0153e50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- 172.21.15.32:0/4012577028 >> 172.21.15.32:0/4012577028 conn(0x7f2bd00cbfe0 msgr2=0x7f2bd00cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- 172.21.15.32:0/4012577028 shutdown_connections 2024-03-16T20:00:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- 172.21.15.32:0/4012577028 wait complete. 2024-03-16T20:00:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 Processor -- start 2024-03-16T20:00:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- start start 2024-03-16T20:00:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd00b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00ae2b0 0x7f2bd014b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd01f2070 con 0x7f2bd006de40 2024-03-16T20:00:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00ae8e0 con 0x7f2bd00ae2b0 2024-03-16T20:00:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd7224640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00aeba0 con 0x7f2bd006e220 2024-03-16T20:00:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd4fc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd4fc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54636/0 (socket says 172.21.15.32:54636) 2024-03-16T20:00:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bd4fc0640 1 -- 172.21.15.32:0/3881604399 learned_addr learned my addr 172.21.15.32:0/3881604399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.192+0000 7f2bcffff640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd00b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.196+0000 7f2bd57c1640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00ae2b0 0x7f2bd014b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.200+0000 7f2bd4fc0640 1 -- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 msgr2=0x7f2bd00b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.200+0000 7f2bd4fc0640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd00b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.200+0000 7f2bd4fc0640 1 -- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00ae2b0 msgr2=0x7f2bd014b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.200+0000 7f2bd4fc0640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00ae2b0 0x7f2bd014b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.200+0000 7f2bd4fc0640 1 -- 172.21.15.32:0/3881604399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc009420 con 0x7f2bd006de40 2024-03-16T20:00:47.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.204+0000 7f2bd4fc0640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c450 tx=0x7f2bc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:47.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4005cf0 con 0x7f2bd006de40 2024-03-16T20:00:47.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4005e50 con 0x7f2bd006de40 2024-03-16T20:00:47.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd01f22a0 con 0x7f2bd006de40 2024-03-16T20:00:47.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4007290 con 0x7f2bd006de40 2024-03-16T20:00:47.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd00af0f0 con 0x7f2bd006de40 2024-03-16T20:00:47.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.208+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd00674e0 con 0x7f2bd006de40 2024-03-16T20:00:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.492+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2bc40073f0 con 0x7f2bd006de40 2024-03-16T20:00:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.492+0000 7f2bcd7fa640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 0x7f2bb4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.492+0000 7f2bcffff640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 0x7f2bb4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.492+0000 7f2bcffff640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 0x7f2bb4069380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b2000 tx=0x7f2bbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:47.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.520+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2bc4054ef0 con 0x7f2bd006de40 2024-03-16T20:00:47.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.520+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2bc4051190 con 0x7f2bd006de40 2024-03-16T20:00:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.728+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bd0068000 con 0x7f2bd006de40 2024-03-16T20:00:47.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.740+0000 7f2bcd7fa640 1 -- 172.21.15.32:0/3881604399 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f2bc4058780 con 0x7f2bd006de40 2024-03-16T20:00:47.715 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:47.716 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 msgr2=0x7f2bb4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 0x7f2bb4069380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b2000 tx=0x7f2bbc008040 comp rx=0 tx=0).stop 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 msgr2=0x7f2bd00afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c450 tx=0x7f2bc400c910 comp rx=0 tx=0).stop 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 shutdown_connections 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2bb4066ec0 0x7f2bb4069380 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bd00ae2b0 0x7f2bd014b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bd006e220 0x7f2bd00b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 --2- 172.21.15.32:0/3881604399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd006de40 0x7f2bd00afbd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 >> 172.21.15.32:0/3881604399 conn(0x7f2bd00cbfe0 msgr2=0x7f2bd006e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 shutdown_connections 2024-03-16T20:00:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:47.744+0000 7f2bd7224640 1 -- 172.21.15.32:0/3881604399 wait complete. 2024-03-16T20:00:48.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:48.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:49.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 Processor -- start 2024-03-16T20:00:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 -- start start 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4cd0151ef0 0x7f4cd00c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4cd00c97b0 0x7f4cd00c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c8720 0x7f4cd015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00b0ad0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd0157cb0 con 0x7f4cd00c97b0 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cd5a87640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd0158c80 con 0x7f4cd00c8720 2024-03-16T20:00:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4ccf7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4cd0151ef0 0x7f4cd00c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4ccf7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4cd0151ef0 0x7f4cd00c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55168/0 (socket says 172.21.15.32:55168) 2024-03-16T20:00:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4ccf7fe640 1 -- 172.21.15.32:0/1144514591 learned_addr learned my addr 172.21.15.32:0/1144514591 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4ccffff640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c8720 0x7f4cd015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.488+0000 7f4cceffd640 1 --2- 172.21.15.32:0/1144514591 >> v2:172.21.15.59:3300/0 conn(0x7f4cd00c97b0 0x7f4cd00c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.492+0000 7f4ccf7fe640 1 -- 172.21.15.32:0/1144514591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c8720 msgr2=0x7f4cd015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.492+0000 7f4ccf7fe640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c8720 0x7f4cd015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.492+0000 7f4ccf7fe640 1 -- 172.21.15.32:0/1144514591 >> v2:172.21.15.59:3300/0 conn(0x7f4cd00c97b0 msgr2=0x7f4cd00c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.492+0000 7f4ccf7fe640 1 --2- 172.21.15.32:0/1144514591 >> v2:172.21.15.59:3300/0 conn(0x7f4cd00c97b0 0x7f4cd00c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.492+0000 7f4ccf7fe640 1 -- 172.21.15.32:0/1144514591 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd015bfc0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.496+0000 7f4ccf7fe640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd00c9270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00cad10 tx=0x7f4cc001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68aaad59180c5a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.500+0000 7f4cce7fc640 1 -- 172.21.15.32:0/1144514591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00a89d0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.500+0000 7f4cce7fc640 1 -- 172.21.15.32:0/1144514591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc00a8b30 con 0x7f4cd0151ef0 2024-03-16T20:00:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.500+0000 7f4cce7fc640 1 -- 172.21.15.32:0/1144514591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00b0440 con 0x7f4cd0151ef0 2024-03-16T20:00:49.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1144514591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 msgr2=0x7f4cd00c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:49.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd00c9270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00cad10 tx=0x7f4cc001ed10 comp rx=0 tx=0).stop 2024-03-16T20:00:49.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1144514591 shutdown_connections 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c8720 0x7f4cd015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1144514591 >> v2:172.21.15.59:3300/0 conn(0x7f4cd00c97b0 0x7f4cd00c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1144514591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd00c9270 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1144514591 >> 172.21.15.32:0/1144514591 conn(0x7f4cd0070f90 msgr2=0x7f4cd00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1144514591 shutdown_connections 2024-03-16T20:00:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1144514591 wait complete. 2024-03-16T20:00:49.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 Processor -- start 2024-03-16T20:00:49.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 -- start start 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd00c8720 0x7f4cd01f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.504+0000 7f4cd5a87640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c97b0 0x7f4cd0158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4cd5a87640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4cd5a87640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01f2e50 con 0x7f4cd0151ef0 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4cd5a87640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00af2b0 con 0x7f4cd00c8720 2024-03-16T20:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4cd5a87640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015b1c0 con 0x7f4cd00c97b0 2024-03-16T20:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4ccf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd00c8720 0x7f4cd01f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4ccffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4cceffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c97b0 0x7f4cd0158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4ccffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55182/0 (socket says 172.21.15.32:55182) 2024-03-16T20:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.508+0000 7f4ccffff640 1 -- 172.21.15.32:0/1671302854 learned_addr learned my addr 172.21.15.32:0/1671302854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.512+0000 7f4ccffff640 1 -- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c97b0 msgr2=0x7f4cd0158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.512+0000 7f4ccffff640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c97b0 0x7f4cd0158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.512+0000 7f4ccffff640 1 -- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd00c8720 msgr2=0x7f4cd01f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.512+0000 7f4ccffff640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd00c8720 0x7f4cd01f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.512+0000 7f4ccffff640 1 -- 172.21.15.32:0/1671302854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc00094d0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.516+0000 7f4ccffff640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800ef00 tx=0x7f4cc800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:49.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.520+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8010040 con 0x7f4cd0151ef0 2024-03-16T20:00:49.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.520+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc8014400 con 0x7f4cd0151ef0 2024-03-16T20:00:49.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.520+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd00b0130 con 0x7f4cd0151ef0 2024-03-16T20:00:49.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.520+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8009ea0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.520+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd00b06b0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.524+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd0068000 con 0x7f4cd0151ef0 2024-03-16T20:00:49.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.548+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4cc80147b0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.548+0000 7f4cb7fff640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 0x7f4cbc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:49.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.548+0000 7f4ccf7fe640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 0x7f4cbc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:49.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.548+0000 7f4ccf7fe640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 0x7f4cbc069330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00b2000 tx=0x7f4cc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:49.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.864+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4cc80563b0 con 0x7f4cd0151ef0 2024-03-16T20:00:49.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:49.864+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4cc8052650 con 0x7f4cd0151ef0 2024-03-16T20:00:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.064+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd0070a40 con 0x7f4cd0151ef0 2024-03-16T20:00:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cb7fff640 1 -- 172.21.15.32:0/1671302854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4cc8018030 con 0x7f4cd0151ef0 2024-03-16T20:00:50.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:50.061 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 msgr2=0x7f4cbc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 0x7f4cbc069330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00b2000 tx=0x7f4cc0009040 comp rx=0 tx=0).stop 2024-03-16T20:00:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 msgr2=0x7f4cd0159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800ef00 tx=0x7f4cc800b480 comp rx=0 tx=0).stop 2024-03-16T20:00:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 shutdown_connections 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4cbc066e70 0x7f4cbc069330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cd0151ef0 0x7f4cd0159c10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4cd00c97b0 0x7f4cd0158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 --2- 172.21.15.32:0/1671302854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd00c8720 0x7f4cd01f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 >> 172.21.15.32:0/1671302854 conn(0x7f4cd0070f90 msgr2=0x7f4cd00b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 shutdown_connections 2024-03-16T20:00:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:50.088+0000 7f4cd5a87640 1 -- 172.21.15.32:0/1671302854 wait complete. 2024-03-16T20:00:50.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:50.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:51.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:51.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.816+0000 7f4753caf640 1 Processor -- start 2024-03-16T20:00:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.816+0000 7f4753caf640 1 -- start start 2024-03-16T20:00:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4753caf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f474c158390 0x7f474c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4753caf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c070760 con 0x7f474c0c6040 2024-03-16T20:00:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4753caf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c0731c0 con 0x7f474c0c6420 2024-03-16T20:00:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4753caf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c0734a0 con 0x7f474c158390 2024-03-16T20:00:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4751a4b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f474c0c6040 0x7f474c157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56370/0 (socket says 172.21.15.32:56370) 2024-03-16T20:00:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f4751a4b640 1 -- 172.21.15.32:0/2296130936 learned_addr learned my addr 172.21.15.32:0/2296130936 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.820+0000 7f475224c640 1 --2- 172.21.15.32:0/2296130936 >> v2:172.21.15.59:3300/0 conn(0x7f474c158390 0x7f474c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.824+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698713327 0 0) 0x7f474c070760 con 0x7f474c0c6040 2024-03-16T20:00:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.824+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4738003530 con 0x7f474c0c6040 2024-03-16T20:00:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.828+0000 7f475224c640 1 -- 172.21.15.32:0/2296130936 >> v1:172.21.15.59:6789/0 conn(0x7f474c0c6420 legacy=0x7f474c1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.828+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3994765401 0 0) 0x7f4738003530 con 0x7f474c0c6040 2024-03-16T20:00:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.828+0000 7f475224c640 1 -- 172.21.15.32:0/2296130936 >> v1:172.21.15.32:6789/0 conn(0x7f474c0c6040 legacy=0x7f474c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.828+0000 7f475224c640 1 -- 172.21.15.32:0/2296130936 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c06fe30 con 0x7f474c158390 2024-03-16T20:00:51.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.840+0000 7f475224c640 1 --2- 172.21.15.32:0/2296130936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c158390 0x7f474c0c8800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4744009d90 tx=0x7f474401c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34749df2d6b2c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748001680 con 0x7f474c158390 2024-03-16T20:00:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47440a56a0 con 0x7f474c158390 2024-03-16T20:00:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4750a49640 1 -- 172.21.15.32:0/2296130936 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47440a5a30 con 0x7f474c158390 2024-03-16T20:00:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 -- 172.21.15.32:0/2296130936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c158390 msgr2=0x7f474c0c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 --2- 172.21.15.32:0/2296130936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c158390 0x7f474c0c8800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4744009d90 tx=0x7f474401c860 comp rx=0 tx=0).stop 2024-03-16T20:00:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 -- 172.21.15.32:0/2296130936 shutdown_connections 2024-03-16T20:00:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 --2- 172.21.15.32:0/2296130936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c158390 0x7f474c0c8800 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 -- 172.21.15.32:0/2296130936 >> 172.21.15.32:0/2296130936 conn(0x7f474c06c0d0 msgr2=0x7f474c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 -- 172.21.15.32:0/2296130936 shutdown_connections 2024-03-16T20:00:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.852+0000 7f4753caf640 1 -- 172.21.15.32:0/2296130936 wait complete. 2024-03-16T20:00:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 Processor -- start 2024-03-16T20:00:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 -- start start 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c0c6420 0x7f474c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f474c158390 0x7f474c1f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c15abb0 con 0x7f474c0c6040 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c154eb0 con 0x7f474c0c6420 2024-03-16T20:00:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4753caf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c15c170 con 0x7f474c158390 2024-03-16T20:00:51.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4751a4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:51.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4751a4b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55192/0 (socket says 172.21.15.32:55192) 2024-03-16T20:00:51.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f4751a4b640 1 -- 172.21.15.32:0/830227225 learned_addr learned my addr 172.21.15.32:0/830227225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:51.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.856+0000 7f475224c640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f474c158390 0x7f474c1f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:51.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.860+0000 7f4751a4b640 1 -- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f474c158390 msgr2=0x7f474c1f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:51.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.860+0000 7f4751a4b640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f474c158390 0x7f474c1f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.860+0000 7f4751a4b640 1 -- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c0c6420 msgr2=0x7f474c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.860+0000 7f4751a4b640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c0c6420 0x7f474c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.860+0000 7f4751a4b640 1 -- 172.21.15.32:0/830227225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4744009a70 con 0x7f474c0c6040 2024-03-16T20:00:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.864+0000 7f4751a4b640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f474800c780 tx=0x7f474800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748005ae0 con 0x7f474c0c6040 2024-03-16T20:00:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748005c40 con 0x7f474c0c6040 2024-03-16T20:00:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748007070 con 0x7f474c0c6040 2024-03-16T20:00:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c15be90 con 0x7f474c0c6040 2024-03-16T20:00:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c154860 con 0x7f474c0c6040 2024-03-16T20:00:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:51.868+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c068000 con 0x7f474c0c6040 2024-03-16T20:00:52.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.156+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f47480071f0 con 0x7f474c0c6040 2024-03-16T20:00:52.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.156+0000 7f47427fc640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 0x7f4730069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:52.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.156+0000 7f475124a640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 0x7f4730069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:52.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.160+0000 7f475124a640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 0x7f4730069350 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f473c000f70 tx=0x7f473c0079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.180+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f47480550e0 con 0x7f474c0c6040 2024-03-16T20:00:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.180+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4748051380 con 0x7f474c0c6040 2024-03-16T20:00:52.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.376+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f474c06fa50 con 0x7f474c0c6040 2024-03-16T20:00:52.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.396+0000 7f47427fc640 1 -- 172.21.15.32:0/830227225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4748017090 con 0x7f474c0c6040 2024-03-16T20:00:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:52.371 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:52.371 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 msgr2=0x7f4730069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 0x7f4730069350 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f473c000f70 tx=0x7f473c0079e0 comp rx=0 tx=0).stop 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 msgr2=0x7f474c152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f474800c780 tx=0x7f474800cc40 comp rx=0 tx=0).stop 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 shutdown_connections 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4730066e90 0x7f4730069350 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:52.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f474c158390 0x7f474c1f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:52.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f474c0c6420 0x7f474c0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:52.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 --2- 172.21.15.32:0/830227225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0c6040 0x7f474c152570 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:52.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 >> 172.21.15.32:0/830227225 conn(0x7f474c06c0d0 msgr2=0x7f474c150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:52.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 shutdown_connections 2024-03-16T20:00:52.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:52.400+0000 7f4753caf640 1 -- 172.21.15.32:0/830227225 wait complete. 2024-03-16T20:00:52.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:52.999 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:54.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 Processor -- start 2024-03-16T20:00:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 -- start start 2024-03-16T20:00:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe25c06f8b0 0x7fe25c151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25c157eb0 con 0x7fe25c0cdf60 2024-03-16T20:00:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25c0c4c80 con 0x7fe25c06f8b0 2024-03-16T20:00:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe260b92640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0c4f40 con 0x7fe25c0cdb80 2024-03-16T20:00:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe25ad9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe25c0cdf60 0x7fe25c158340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56384/0 (socket says 172.21.15.32:56384) 2024-03-16T20:00:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe25ad9d640 1 -- 172.21.15.32:0/3801840974 learned_addr learned my addr 172.21.15.32:0/3801840974 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe259d9b640 1 --2- 172.21.15.32:0/3801840974 >> v2:172.21.15.32:3300/0 conn(0x7fe25c06f8b0 0x7fe25c151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.136+0000 7fe25a59c640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.140+0000 7fe259d9b640 1 -- 172.21.15.32:0/3801840974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 msgr2=0x7fe25c06f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:54.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.140+0000 7fe259d9b640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c06f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.140+0000 7fe259d9b640 1 -- 172.21.15.32:0/3801840974 >> v1:172.21.15.32:6789/0 conn(0x7fe25c0cdf60 legacy=0x7fe25c158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.140+0000 7fe259d9b640 1 -- 172.21.15.32:0/3801840974 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c0c5220 con 0x7fe25c06f8b0 2024-03-16T20:00:54.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.148+0000 7fe259d9b640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c06f8b0 0x7fe25c151b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe2500097c0 tx=0x7fe25001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451f9b2b278b28ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:54.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.152+0000 7fe25959a640 1 -- 172.21.15.32:0/3801840974 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500aa5f0 con 0x7fe25c06f8b0 2024-03-16T20:00:54.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.152+0000 7fe25959a640 1 -- 172.21.15.32:0/3801840974 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2500aa750 con 0x7fe25c06f8b0 2024-03-16T20:00:54.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.152+0000 7fe25959a640 1 -- 172.21.15.32:0/3801840974 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500aaae0 con 0x7fe25c06f8b0 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- 172.21.15.32:0/3801840974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c06f8b0 msgr2=0x7fe25c151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c06f8b0 0x7fe25c151b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe2500097c0 tx=0x7fe25001c470 comp rx=0 tx=0).stop 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- 172.21.15.32:0/3801840974 shutdown_connections 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c06f8b0 0x7fe25c151b20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- 172.21.15.32:0/3801840974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c06f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- 172.21.15.32:0/3801840974 >> 172.21.15.32:0/3801840974 conn(0x7fe25c066ea0 msgr2=0x7fe25c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:54.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- 172.21.15.32:0/3801840974 shutdown_connections 2024-03-16T20:00:54.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- 172.21.15.32:0/3801840974 wait complete. 2024-03-16T20:00:54.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 Processor -- start 2024-03-16T20:00:54.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- start start 2024-03-16T20:00:54.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c145550 0x7fe25c155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c149ad0 con 0x7fe25c0cdf60 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c1495f0 con 0x7fe25c145550 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe260b92640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0cb840 con 0x7fe25c0cdb80 2024-03-16T20:00:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe259d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe259d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55218/0 (socket says 172.21.15.32:55218) 2024-03-16T20:00:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.156+0000 7fe259d9b640 1 -- 172.21.15.32:0/2203930432 learned_addr learned my addr 172.21.15.32:0/2203930432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.160+0000 7fe25ad9d640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c145550 0x7fe25c155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.160+0000 7fe25a59c640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.164+0000 7fe259d9b640 1 -- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 msgr2=0x7fe25c146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.164+0000 7fe259d9b640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c146280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.164+0000 7fe259d9b640 1 -- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c145550 msgr2=0x7fe25c155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.164+0000 7fe259d9b640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c145550 0x7fe25c155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.164+0000 7fe259d9b640 1 -- 172.21.15.32:0/2203930432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250009420 con 0x7fe25c0cdf60 2024-03-16T20:00:54.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.168+0000 7fe259d9b640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe25001a0a0 tx=0x7fe25001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:54.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500be610 con 0x7fe25c0cdf60 2024-03-16T20:00:54.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2500be790 con 0x7fe25c0cdf60 2024-03-16T20:00:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe260b92640 1 -- 172.21.15.32:0/2203930432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c0cbb20 con 0x7fe25c0cdf60 2024-03-16T20:00:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500beb40 con 0x7fe25c0cdf60 2024-03-16T20:00:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe260b92640 1 -- 172.21.15.32:0/2203930432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c0cc0e0 con 0x7fe25c0cdf60 2024-03-16T20:00:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.172+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c145960 con 0x7fe25c0cdf60 2024-03-16T20:00:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.212+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe2500b4400 con 0x7fe25c0cdf60 2024-03-16T20:00:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.212+0000 7fe24affd640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 0x7fe23c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.212+0000 7fe25a59c640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 0x7fe23c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:54.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.212+0000 7fe25a59c640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 0x7fe23c069300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009b60 tx=0x7fe24c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.496+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe2500a7070 con 0x7fe25c0cdf60 2024-03-16T20:00:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.496+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe2500eca60 con 0x7fe25c0cdf60 2024-03-16T20:00:54.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.712+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe25c145960 con 0x7fe25c0cdf60 2024-03-16T20:00:54.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe24affd640 1 -- 172.21.15.32:0/2203930432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe25c145960 con 0x7fe25c0cdf60 2024-03-16T20:00:54.796 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:54.796 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:54.797 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 msgr2=0x7fe23c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 0x7fe23c069300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009b60 tx=0x7fe24c009190 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 msgr2=0x7fe25c145010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe25001a0a0 tx=0x7fe25001acb0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 shutdown_connections 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe23c066e40 0x7fe23c069300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c145550 0x7fe25c155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe25c0cdf60 0x7fe25c145010 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 --2- 172.21.15.32:0/2203930432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe25c0cdb80 0x7fe25c146280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 >> 172.21.15.32:0/2203930432 conn(0x7fe25c066ea0 msgr2=0x7fe25c158340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 shutdown_connections 2024-03-16T20:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:54.824+0000 7fe248ff9640 1 -- 172.21.15.32:0/2203930432 wait complete. 2024-03-16T20:00:55.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:55.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:56.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 Processor -- start 2024-03-16T20:00:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 -- start start 2024-03-16T20:00:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 0x7fcb8c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb8c0c97b0 0x7fcb8c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c158c80 con 0x7fcb8c0c8720 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b0ad0 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb91154640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c157cb0 con 0x7fcb8c151ef0 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb8ad9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 0x7fcb8c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.564+0000 7fcb8a59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb8c0c97b0 0x7fcb8c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.568+0000 7fcb8b59e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb8c0c8720 0x7fcb8c15b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56398/0 (socket says 172.21.15.32:56398) 2024-03-16T20:00:56.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.568+0000 7fcb8b59e640 1 -- 172.21.15.32:0/120448628 learned_addr learned my addr 172.21.15.32:0/120448628 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:00:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.572+0000 7fcb8ad9d640 1 -- 172.21.15.32:0/120448628 >> v2:172.21.15.32:3300/0 conn(0x7fcb8c0c97b0 msgr2=0x7fcb8c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.572+0000 7fcb8ad9d640 1 --2- 172.21.15.32:0/120448628 >> v2:172.21.15.32:3300/0 conn(0x7fcb8c0c97b0 0x7fcb8c0c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.572+0000 7fcb8ad9d640 1 -- 172.21.15.32:0/120448628 >> v1:172.21.15.32:6789/0 conn(0x7fcb8c0c8720 legacy=0x7fcb8c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.572+0000 7fcb8ad9d640 1 -- 172.21.15.32:0/120448628 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c15bfc0 con 0x7fcb8c151ef0 2024-03-16T20:00:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.572+0000 7fcb8ad9d640 1 --2- 172.21.15.32:0/120448628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 0x7fcb8c0c9270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0097c0 tx=0x7fcb7c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c4b9e46ffdbccda server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb89d9b640 1 -- 172.21.15.32:0/120448628 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c0a8990 con 0x7fcb8c151ef0 2024-03-16T20:00:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb89d9b640 1 -- 172.21.15.32:0/120448628 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c0a8af0 con 0x7fcb8c151ef0 2024-03-16T20:00:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb89d9b640 1 -- 172.21.15.32:0/120448628 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c0a8e80 con 0x7fcb8c151ef0 2024-03-16T20:00:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- 172.21.15.32:0/120448628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 msgr2=0x7fcb8c0c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 --2- 172.21.15.32:0/120448628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 0x7fcb8c0c9270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0097c0 tx=0x7fcb7c01ec90 comp rx=0 tx=0).stop 2024-03-16T20:00:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- 172.21.15.32:0/120448628 shutdown_connections 2024-03-16T20:00:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 --2- 172.21.15.32:0/120448628 >> v2:172.21.15.32:3300/0 conn(0x7fcb8c0c97b0 0x7fcb8c0c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 --2- 172.21.15.32:0/120448628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c151ef0 0x7fcb8c0c9270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:56.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- 172.21.15.32:0/120448628 >> 172.21.15.32:0/120448628 conn(0x7fcb8c070f90 msgr2=0x7fcb8c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:56.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- 172.21.15.32:0/120448628 shutdown_connections 2024-03-16T20:00:56.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- 172.21.15.32:0/120448628 wait complete. 2024-03-16T20:00:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 Processor -- start 2024-03-16T20:00:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.584+0000 7fcb91154640 1 -- start start 2024-03-16T20:00:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0c8720 0x7fcb8c1590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb8c151ef0 0x7fcb8c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c15b130 con 0x7fcb8c151ef0 2024-03-16T20:00:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0cc8f0 con 0x7fcb8c0c8720 2024-03-16T20:00:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb91154640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b13a0 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb8a59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb8a59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:41354/0 (socket says 172.21.15.32:41354) 2024-03-16T20:00:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb8a59c640 1 -- 172.21.15.32:0/1839704270 learned_addr learned my addr 172.21.15.32:0/1839704270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.588+0000 7fcb8ad9d640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0c8720 0x7fcb8c1590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8b59e640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb8c151ef0 0x7fcb8c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8a59c640 1 -- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0c8720 msgr2=0x7fcb8c1590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8a59c640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0c8720 0x7fcb8c1590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8a59c640 1 -- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb8c151ef0 msgr2=0x7fcb8c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8a59c640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb8c151ef0 0x7fcb8c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.596+0000 7fcb8a59c640 1 -- 172.21.15.32:0/1839704270 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c009420 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.604+0000 7fcb8a59c640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e930 tx=0x7fcb8000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80009690 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8000ca00 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000cdb0 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c0b0290 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c06da40 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.608+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c068000 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.944+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcb80013470 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.944+0000 7fcb737fe640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 0x7fcb78069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:56.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.944+0000 7fcb8ad9d640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 0x7fcb78069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:56.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.948+0000 7fcb8ad9d640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 0x7fcb78069330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b2000 tx=0x7fcb7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:56.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.948+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fcb80055f40 con 0x7fcb8c0c97b0 2024-03-16T20:00:56.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:56.968+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fcb800521e0 con 0x7fcb8c0c97b0 2024-03-16T20:00:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.192+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb8c0b0290 con 0x7fcb8c0c97b0 2024-03-16T20:00:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.376+0000 7fcb737fe640 1 -- 172.21.15.32:0/1839704270 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fcb8c0b0290 con 0x7fcb8c0c97b0 2024-03-16T20:00:57.349 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:57.349 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:57.350 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.376+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 msgr2=0x7fcb78069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.376+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 0x7fcb78069330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b2000 tx=0x7fcb7c008040 comp rx=0 tx=0).stop 2024-03-16T20:00:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 msgr2=0x7fcb8c0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e930 tx=0x7fcb8000edf0 comp rx=0 tx=0).stop 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 shutdown_connections 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcb78066e70 0x7fcb78069330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb8c151ef0 0x7fcb8c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcb8c0c97b0 0x7fcb8c0ccd10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 --2- 172.21.15.32:0/1839704270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0c8720 0x7fcb8c1590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 >> 172.21.15.32:0/1839704270 conn(0x7fcb8c070f90 msgr2=0x7fcb8c06efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 shutdown_connections 2024-03-16T20:00:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:57.380+0000 7fcb91154640 1 -- 172.21.15.32:0/1839704270 wait complete. 2024-03-16T20:00:57.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:00:57.978 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:00:58.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:00:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.124+0000 7ff00e592640 1 Processor -- start 2024-03-16T20:00:59.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.124+0000 7ff00e592640 1 -- start start 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00e592640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff00806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00e592640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff008072a10 0x7ff008072e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00e592640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00806bae0 con 0x7ff0080c2d70 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00e592640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00815a540 con 0x7ff008072a10 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00e592640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080c7fd0 con 0x7ff008154350 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff008072a10 0x7ff008072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff008072a10 0x7ff008072e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39874/0 (socket says 172.21.15.32:39874) 2024-03-16T20:00:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 -- 172.21.15.32:0/2939416489 learned_addr learned my addr 172.21.15.32:0/2939416489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:59.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff007fff640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff00806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 -- 172.21.15.32:0/2939416489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 msgr2=0x7ff00806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:00:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff00806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 -- 172.21.15.32:0/2939416489 >> v1:172.21.15.59:6789/0 conn(0x7ff0080c2d70 legacy=0x7ff00815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.128+0000 7ff00cb2f640 1 -- 172.21.15.32:0/2939416489 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00806bd10 con 0x7ff008072a10 2024-03-16T20:00:59.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.136+0000 7ff00cb2f640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff008072a10 0x7ff008072e60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff000009e50 tx=0x7ff00001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc10805b4e77c1a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff006ffd640 1 -- 172.21.15.32:0/2939416489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0000a5660 con 0x7ff008072a10 2024-03-16T20:00:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff006ffd640 1 -- 172.21.15.32:0/2939416489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0000a57c0 con 0x7ff008072a10 2024-03-16T20:00:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff006ffd640 1 -- 172.21.15.32:0/2939416489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0000a5b50 con 0x7ff008072a10 2024-03-16T20:00:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- 172.21.15.32:0/2939416489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff008072a10 msgr2=0x7ff008072e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff008072a10 0x7ff008072e60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff000009e50 tx=0x7ff00001c9a0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- 172.21.15.32:0/2939416489 shutdown_connections 2024-03-16T20:00:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff008072a10 0x7ff008072e60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 --2- 172.21.15.32:0/2939416489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff00806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- 172.21.15.32:0/2939416489 >> 172.21.15.32:0/2939416489 conn(0x7ff00806afc0 msgr2=0x7ff00806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- 172.21.15.32:0/2939416489 shutdown_connections 2024-03-16T20:00:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- 172.21.15.32:0/2939416489 wait complete. 2024-03-16T20:00:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 Processor -- start 2024-03-16T20:00:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.140+0000 7ff00e592640 1 -- start start 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff008072a10 0x7ff00815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff0081f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0081f6a70 con 0x7ff0080c2d70 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080c79c0 con 0x7ff008072a10 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00e592640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0081f5b80 con 0x7ff008154350 2024-03-16T20:00:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff0077fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff0077fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39880/0 (socket says 172.21.15.32:39880) 2024-03-16T20:00:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff0077fe640 1 -- 172.21.15.32:0/1937370651 learned_addr learned my addr 172.21.15.32:0/1937370651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:00:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff007fff640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff008072a10 0x7ff00815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.144+0000 7ff00cb2f640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff0081f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.148+0000 7ff0077fe640 1 -- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 msgr2=0x7ff0081f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.148+0000 7ff0077fe640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff0081f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.148+0000 7ff0077fe640 1 -- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff008072a10 msgr2=0x7ff00815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.148+0000 7ff0077fe640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff008072a10 0x7ff00815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.148+0000 7ff0077fe640 1 -- 172.21.15.32:0/1937370651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff000009b30 con 0x7ff0080c2d70 2024-03-16T20:00:59.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.152+0000 7ff0077fe640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feffc00c420 tx=0x7feffc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc0052d0 con 0x7ff0080c2d70 2024-03-16T20:00:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc005430 con 0x7ff0080c2d70 2024-03-16T20:00:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0081f5e60 con 0x7ff0080c2d70 2024-03-16T20:00:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc007260 con 0x7ff0080c2d70 2024-03-16T20:00:59.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0081f6420 con 0x7ff0080c2d70 2024-03-16T20:00:59.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.156+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0080674e0 con 0x7ff0080c2d70 2024-03-16T20:00:59.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.192+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feffc0073c0 con 0x7ff0080c2d70 2024-03-16T20:00:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.192+0000 7ff004ff9640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 0x7feff4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:00:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.192+0000 7ff007fff640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 0x7feff4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:00:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.192+0000 7ff007fff640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 0x7feff4069300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feff8009ac0 tx=0x7feff8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:00:59.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.476+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7feffc007670 con 0x7ff0080c2d70 2024-03-16T20:00:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.476+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7feffc0514c0 con 0x7ff0080c2d70 2024-03-16T20:00:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.728+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff00806e930 con 0x7ff0080c2d70 2024-03-16T20:00:59.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.808+0000 7ff004ff9640 1 -- 172.21.15.32:0/1937370651 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7feffc020020 con 0x7ff0080c2d70 2024-03-16T20:00:59.784 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:00:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:00:59.785 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:00:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 msgr2=0x7feff4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 0x7feff4069300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feff8009ac0 tx=0x7feff8009190 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 msgr2=0x7ff008158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feffc00c420 tx=0x7feffc00c8e0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 shutdown_connections 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7feff4066e40 0x7feff4069300 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff008154350 0x7ff0081f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0080c2d70 0x7ff008158190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 --2- 172.21.15.32:0/1937370651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff008072a10 0x7ff00815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 >> 172.21.15.32:0/1937370651 conn(0x7ff00806afc0 msgr2=0x7ff0080c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 shutdown_connections 2024-03-16T20:00:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:00:59.812+0000 7ff00e592640 1 -- 172.21.15.32:0/1937370651 wait complete. 2024-03-16T20:01:00.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:00.408 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:01.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.544+0000 7fb5a6ed3640 1 Processor -- start 2024-03-16T20:01:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.544+0000 7fb5a6ed3640 1 -- start start 2024-03-16T20:01:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb5a6ed3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5a015b9b0 0x7fb5a00c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb5a6ed3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0158c80 con 0x7fb5a00c81e0 2024-03-16T20:01:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb5a6ed3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a015bfc0 con 0x7fb5a0151ef0 2024-03-16T20:01:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb5a6ed3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00b0ad0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb59ffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb5a00c81e0 0x7fb5a015b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37764/0 (socket says 172.21.15.32:37764) 2024-03-16T20:01:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb59ffff640 1 -- 172.21.15.32:0/1142511305 learned_addr learned my addr 172.21.15.32:0/1142511305 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.548+0000 7fb5a5470640 1 --2- 172.21.15.32:0/1142511305 >> v2:172.21.15.59:3300/0 conn(0x7fb5a015b9b0 0x7fb5a00c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.552+0000 7fb59f7fe640 1 -- 172.21.15.32:0/1142511305 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669490040 0 0) 0x7fb5a0158c80 con 0x7fb5a00c81e0 2024-03-16T20:01:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.552+0000 7fb59f7fe640 1 -- 172.21.15.32:0/1142511305 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb590003530 con 0x7fb5a00c81e0 2024-03-16T20:01:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.552+0000 7fb5a5470640 1 -- 172.21.15.32:0/1142511305 >> v1:172.21.15.59:6789/0 conn(0x7fb5a0151ef0 legacy=0x7fb5a00c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.552+0000 7fb5a5470640 1 -- 172.21.15.32:0/1142511305 >> v1:172.21.15.32:6789/0 conn(0x7fb5a00c81e0 legacy=0x7fb5a015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.552+0000 7fb5a5470640 1 -- 172.21.15.32:0/1142511305 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0157ec0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.556+0000 7fb5a5470640 1 --2- 172.21.15.32:0/1142511305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a015b9b0 0x7fb5a00c8630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb598009d90 tx=0x7fb59801f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50a642ae642feb44 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.560+0000 7fb59f7fe640 1 -- 172.21.15.32:0/1142511305 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5980af600 con 0x7fb5a015b9b0 2024-03-16T20:01:01.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb59f7fe640 1 -- 172.21.15.32:0/1142511305 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5980ab050 con 0x7fb5a015b9b0 2024-03-16T20:01:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb59f7fe640 1 -- 172.21.15.32:0/1142511305 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5980af960 con 0x7fb5a015b9b0 2024-03-16T20:01:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/1142511305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a015b9b0 msgr2=0x7fb5a00c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/1142511305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a015b9b0 0x7fb5a00c8630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb598009d90 tx=0x7fb59801f120 comp rx=0 tx=0).stop 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/1142511305 shutdown_connections 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/1142511305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a015b9b0 0x7fb5a00c8630 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/1142511305 >> 172.21.15.32:0/1142511305 conn(0x7fb5a0070f90 msgr2=0x7fb5a00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/1142511305 shutdown_connections 2024-03-16T20:01:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/1142511305 wait complete. 2024-03-16T20:01:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 Processor -- start 2024-03-16T20:01:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.568+0000 7fb5a6ed3640 1 -- start start 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5a00c81e0 0x7fb5a0155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a0151ef0 0x7fb5a0156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a014c0d0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00cd4a0 con 0x7fb5a0151ef0 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a6ed3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01ee470 con 0x7fb5a00c81e0 2024-03-16T20:01:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a5470640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:01.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a5470640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39892/0 (socket says 172.21.15.32:39892) 2024-03-16T20:01:01.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb59ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a0151ef0 0x7fb5a0156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:01.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a5470640 1 -- 172.21.15.32:0/2989110512 learned_addr learned my addr 172.21.15.32:0/2989110512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:01.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.572+0000 7fb5a4c6f640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5a00c81e0 0x7fb5a0155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 -- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5a00c81e0 msgr2=0x7fb5a0155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5a00c81e0 0x7fb5a0155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 -- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a0151ef0 msgr2=0x7fb5a0156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a0151ef0 0x7fb5a0156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 -- 172.21.15.32:0/2989110512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009a70 con 0x7fb5a015b9b0 2024-03-16T20:01:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.576+0000 7fb5a5470640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb59801bd30 tx=0x7fb59801bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:01.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598004250 con 0x7fb5a015b9b0 2024-03-16T20:01:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5980043b0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a01ee750 con 0x7fb5a015b9b0 2024-03-16T20:01:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598004720 con 0x7fb5a015b9b0 2024-03-16T20:01:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a00cd1c0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.584+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0068000 con 0x7fb5a015b9b0 2024-03-16T20:01:01.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.872+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb5980ba420 con 0x7fb5a015b9b0 2024-03-16T20:01:01.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.872+0000 7fb59d7fa640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 0x7fb584069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:01.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.872+0000 7fb5a4c6f640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 0x7fb584069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:01.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.872+0000 7fb5a4c6f640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 0x7fb584069300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb594007fd0 tx=0x7fb594009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:01.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.912+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb5980f42a0 con 0x7fb5a015b9b0 2024-03-16T20:01:01.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:01.912+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb5980f0540 con 0x7fb5a015b9b0 2024-03-16T20:01:02.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.212+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a0070a40 con 0x7fb5a015b9b0 2024-03-16T20:01:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.236+0000 7fb59d7fa640 1 -- 172.21.15.32:0/2989110512 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb5980f3c00 con 0x7fb5a015b9b0 2024-03-16T20:01:02.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:02.210 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 msgr2=0x7fb584069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 0x7fb584069300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb594007fd0 tx=0x7fb594009040 comp rx=0 tx=0).stop 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 msgr2=0x7fb5a0157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb59801bd30 tx=0x7fb59801bec0 comp rx=0 tx=0).stop 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 shutdown_connections 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb584066e40 0x7fb584069300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a015b9b0 0x7fb5a0157000 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a0151ef0 0x7fb5a0156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 --2- 172.21.15.32:0/2989110512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5a00c81e0 0x7fb5a0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 >> 172.21.15.32:0/2989110512 conn(0x7fb5a0070f90 msgr2=0x7fb5a00c3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 shutdown_connections 2024-03-16T20:01:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:02.240+0000 7fb5a6ed3640 1 -- 172.21.15.32:0/2989110512 wait complete. 2024-03-16T20:01:02.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:02.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:03.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:03.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 Processor -- start 2024-03-16T20:01:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 -- start start 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4440c81e0 0x7fc4440c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc44415b2a0 0x7fc44415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc444158c80 con 0x7fc444151ef0 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4440b0ad0 con 0x7fc44415b2a0 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc44b73d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc444157cb0 con 0x7fc4440c81e0 2024-03-16T20:01:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc449cda640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc44415b2a0 0x7fc44415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc449cda640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc44415b2a0 0x7fc44415b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39906/0 (socket says 172.21.15.32:39906) 2024-03-16T20:01:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.968+0000 7fc449cda640 1 -- 172.21.15.32:0/3550173422 learned_addr learned my addr 172.21.15.32:0/3550173422 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.972+0000 7fc448cd8640 1 --2- 172.21.15.32:0/3550173422 >> v2:172.21.15.59:3300/0 conn(0x7fc4440c81e0 0x7fc4440c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.972+0000 7fc449cda640 1 -- 172.21.15.32:0/3550173422 >> v2:172.21.15.59:3300/0 conn(0x7fc4440c81e0 msgr2=0x7fc4440c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.972+0000 7fc449cda640 1 --2- 172.21.15.32:0/3550173422 >> v2:172.21.15.59:3300/0 conn(0x7fc4440c81e0 0x7fc4440c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.972+0000 7fc449cda640 1 -- 172.21.15.32:0/3550173422 >> v1:172.21.15.32:6789/0 conn(0x7fc444151ef0 legacy=0x7fc4440c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.972+0000 7fc449cda640 1 -- 172.21.15.32:0/3550173422 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44415bfc0 con 0x7fc44415b2a0 2024-03-16T20:01:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.976+0000 7fc449cda640 1 --2- 172.21.15.32:0/3550173422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc44415b2a0 0x7fc44415b6a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4440cad10 tx=0x7fc43c01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d03829d0edb6dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc43bfff640 1 -- 172.21.15.32:0/3550173422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0af400 con 0x7fc44415b2a0 2024-03-16T20:01:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc43bfff640 1 -- 172.21.15.32:0/3550173422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c0ab050 con 0x7fc44415b2a0 2024-03-16T20:01:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc43bfff640 1 -- 172.21.15.32:0/3550173422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0af780 con 0x7fc44415b2a0 2024-03-16T20:01:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- 172.21.15.32:0/3550173422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc44415b2a0 msgr2=0x7fc44415b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 --2- 172.21.15.32:0/3550173422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc44415b2a0 0x7fc44415b6a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4440cad10 tx=0x7fc43c01f1a0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- 172.21.15.32:0/3550173422 shutdown_connections 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 --2- 172.21.15.32:0/3550173422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc44415b2a0 0x7fc44415b6a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 --2- 172.21.15.32:0/3550173422 >> v2:172.21.15.59:3300/0 conn(0x7fc4440c81e0 0x7fc4440c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- 172.21.15.32:0/3550173422 >> 172.21.15.32:0/3550173422 conn(0x7fc444070f90 msgr2=0x7fc4440713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- 172.21.15.32:0/3550173422 shutdown_connections 2024-03-16T20:01:03.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- 172.21.15.32:0/3550173422 wait complete. 2024-03-16T20:01:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 Processor -- start 2024-03-16T20:01:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.984+0000 7fc44b73d640 1 -- start start 2024-03-16T20:01:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4440c81e0 0x7fc44406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc44415b2a0 0x7fc4440b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440b1f20 con 0x7fc444151ef0 2024-03-16T20:01:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44414d680 con 0x7fc44415b2a0 2024-03-16T20:01:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc44b73d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44406c7e0 con 0x7fc4440c81e0 2024-03-16T20:01:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc448cd8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc448cd8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39916/0 (socket says 172.21.15.32:39916) 2024-03-16T20:01:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc448cd8640 1 -- 172.21.15.32:0/191805540 learned_addr learned my addr 172.21.15.32:0/191805540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc449cda640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc44415b2a0 0x7fc4440b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.988+0000 7fc4494d9640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4440c81e0 0x7fc44406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 -- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4440c81e0 msgr2=0x7fc44406d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4440c81e0 0x7fc44406d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 -- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc44415b2a0 msgr2=0x7fc4440b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc44415b2a0 0x7fc4440b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 -- 172.21.15.32:0/191805540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c009b30 con 0x7fc444151ef0 2024-03-16T20:01:03.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:03.992+0000 7fc448cd8640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc43400e930 tx=0x7fc43400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:03.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434009690 con 0x7fc444151ef0 2024-03-16T20:01:03.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43400ca00 con 0x7fc444151ef0 2024-03-16T20:01:03.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44406ca50 con 0x7fc444151ef0 2024-03-16T20:01:03.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43400cd70 con 0x7fc444151ef0 2024-03-16T20:01:03.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44414dbb0 con 0x7fc444151ef0 2024-03-16T20:01:03.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.000+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc444068000 con 0x7fc444151ef0 2024-03-16T20:01:03.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.024+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc434013470 con 0x7fc444151ef0 2024-03-16T20:01:03.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.024+0000 7fc439ffb640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 0x7fc428069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:03.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.024+0000 7fc4494d9640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 0x7fc428069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:03.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.024+0000 7fc4494d9640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 0x7fc428069300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc440007fd0 tx=0x7fc440007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:04.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.304+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc4340560f0 con 0x7fc444151ef0 2024-03-16T20:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.304+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc434052390 con 0x7fc444151ef0 2024-03-16T20:01:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.552+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc444070a40 con 0x7fc444151ef0 2024-03-16T20:01:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc439ffb640 1 -- 172.21.15.32:0/191805540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc434021070 con 0x7fc444151ef0 2024-03-16T20:01:04.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:04.550 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 msgr2=0x7fc428069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 0x7fc428069300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc440007fd0 tx=0x7fc440007e40 comp rx=0 tx=0).stop 2024-03-16T20:01:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 msgr2=0x7fc4440b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc43400e930 tx=0x7fc43400edf0 comp rx=0 tx=0).stop 2024-03-16T20:01:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 shutdown_connections 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc428066e40 0x7fc428069300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc44415b2a0 0x7fc4440b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc444151ef0 0x7fc4440b2720 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 --2- 172.21.15.32:0/191805540 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4440c81e0 0x7fc44406d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.576+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 >> 172.21.15.32:0/191805540 conn(0x7fc444070f90 msgr2=0x7fc444156080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.580+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 shutdown_connections 2024-03-16T20:01:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:04.580+0000 7fc44b73d640 1 -- 172.21.15.32:0/191805540 wait complete. 2024-03-16T20:01:05.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:05.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:06.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:06.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 Processor -- start 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 -- start start 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4904151ef0 0x7f49040c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 0x7f49040c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904158c80 con 0x7f49040c8720 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49040b0ad0 con 0x7f49040c97b0 2024-03-16T20:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4909f18640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4904157cb0 con 0x7f4904151ef0 2024-03-16T20:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4902ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 0x7f49040c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f49037fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4904151ef0 0x7f49040c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4902ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 0x7f49040c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39924/0 (socket says 172.21.15.32:39924) 2024-03-16T20:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.312+0000 7f4902ffd640 1 -- 172.21.15.32:0/3133273085 learned_addr learned my addr 172.21.15.32:0/3133273085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.320+0000 7f49037fe640 1 -- 172.21.15.32:0/3133273085 >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 msgr2=0x7f49040c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.320+0000 7f49037fe640 1 --2- 172.21.15.32:0/3133273085 >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 0x7f49040c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.320+0000 7f49037fe640 1 -- 172.21.15.32:0/3133273085 >> v1:172.21.15.32:6789/0 conn(0x7f49040c8720 legacy=0x7f490415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.320+0000 7f49037fe640 1 -- 172.21.15.32:0/3133273085 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490415bfc0 con 0x7f4904151ef0 2024-03-16T20:01:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.324+0000 7f49037fe640 1 --2- 172.21.15.32:0/3133273085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4904151ef0 0x7f49040c9270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49040cb3a0 tx=0x7f48f401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99fb8cee3bd731ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f49027fc640 1 -- 172.21.15.32:0/3133273085 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40a8970 con 0x7f4904151ef0 2024-03-16T20:01:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f49027fc640 1 -- 172.21.15.32:0/3133273085 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f40a8ad0 con 0x7f4904151ef0 2024-03-16T20:01:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f49027fc640 1 -- 172.21.15.32:0/3133273085 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40a8e60 con 0x7f4904151ef0 2024-03-16T20:01:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- 172.21.15.32:0/3133273085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4904151ef0 msgr2=0x7f49040c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 --2- 172.21.15.32:0/3133273085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4904151ef0 0x7f49040c9270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49040cb3a0 tx=0x7f48f401ec70 comp rx=0 tx=0).stop 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- 172.21.15.32:0/3133273085 shutdown_connections 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 --2- 172.21.15.32:0/3133273085 >> v2:172.21.15.32:3300/0 conn(0x7f49040c97b0 0x7f49040c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 --2- 172.21.15.32:0/3133273085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4904151ef0 0x7f49040c9270 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- 172.21.15.32:0/3133273085 >> 172.21.15.32:0/3133273085 conn(0x7f4904070f90 msgr2=0x7f49040713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- 172.21.15.32:0/3133273085 shutdown_connections 2024-03-16T20:01:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- 172.21.15.32:0/3133273085 wait complete. 2024-03-16T20:01:06.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 Processor -- start 2024-03-16T20:01:06.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.336+0000 7f4909f18640 1 -- start start 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49040c8720 0x7f49040b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49040c97b0 0x7f49041ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041ed750 con 0x7f4904151ef0 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040c55e0 con 0x7f49040c8720 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4909f18640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49041f1850 con 0x7f49040c97b0 2024-03-16T20:01:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4903fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4903fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39936/0 (socket says 172.21.15.32:39936) 2024-03-16T20:01:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4903fff640 1 -- 172.21.15.32:0/3017730109 learned_addr learned my addr 172.21.15.32:0/3017730109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f4902ffd640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49040c97b0 0x7f49041ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.340+0000 7f49037fe640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49040c8720 0x7f49040b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.344+0000 7f4903fff640 1 -- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49040c97b0 msgr2=0x7f49041ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.344+0000 7f4903fff640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49040c97b0 0x7f49041ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.344+0000 7f4903fff640 1 -- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49040c8720 msgr2=0x7f49040b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.344+0000 7f4903fff640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49040c8720 0x7f49040b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.344+0000 7f4903fff640 1 -- 172.21.15.32:0/3017730109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f4009420 con 0x7f4904151ef0 2024-03-16T20:01:06.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.348+0000 7f4903fff640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c770 tx=0x7f48fc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.352+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc003b80 con 0x7f4904151ef0 2024-03-16T20:01:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.352+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc003ce0 con 0x7f4904151ef0 2024-03-16T20:01:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.352+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc00aa90 con 0x7f4904151ef0 2024-03-16T20:01:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.352+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49041ee6a0 con 0x7f4904151ef0 2024-03-16T20:01:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.352+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49040b2a50 con 0x7f4904151ef0 2024-03-16T20:01:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.356+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904068000 con 0x7f4904151ef0 2024-03-16T20:01:06.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.388+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f48fc00abf0 con 0x7f4904151ef0 2024-03-16T20:01:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.388+0000 7f48ebfff640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 0x7f48f0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.388+0000 7f49037fe640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 0x7f48f0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.388+0000 7f49037fe640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 0x7f48f0069380 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f48f40b2000 tx=0x7f48f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:06.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.680+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f48fc055de0 con 0x7f4904151ef0 2024-03-16T20:01:06.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.680+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f48fc052080 con 0x7f4904151ef0 2024-03-16T20:01:06.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.880+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4904070a40 con 0x7f4904151ef0 2024-03-16T20:01:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.964+0000 7f48ebfff640 1 -- 172.21.15.32:0/3017730109 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f48fc055740 con 0x7f4904151ef0 2024-03-16T20:01:06.940 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:06.940 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 msgr2=0x7f48f0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 0x7f48f0069380 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f48f40b2000 tx=0x7f48f4008040 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 msgr2=0x7f490414ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c770 tx=0x7f48fc00cc30 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 shutdown_connections 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48f0066ec0 0x7f48f0069380 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4904151ef0 0x7f490414ff00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49040c97b0 0x7f49041ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 --2- 172.21.15.32:0/3017730109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49040c8720 0x7f49040b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 >> 172.21.15.32:0/3017730109 conn(0x7f4904070f90 msgr2=0x7f4904071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 shutdown_connections 2024-03-16T20:01:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:06.968+0000 7f4909f18640 1 -- 172.21.15.32:0/3017730109 wait complete. 2024-03-16T20:01:07.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:07.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:08.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:08.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.712+0000 7fa79727f640 1 Processor -- start 2024-03-16T20:01:08.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.712+0000 7fa79727f640 1 -- start start 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79727f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa790151ef0 0x7fa7900c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79727f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7900c97b0 0x7fa7900c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79727f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa790158c80 con 0x7fa7900c8720 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79727f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7900b0ad0 con 0x7fa7900c97b0 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79727f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa790157cb0 con 0x7fa790151ef0 2024-03-16T20:01:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79481a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7900c97b0 0x7fa7900c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79501b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa790151ef0 0x7fa7900c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79481a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7900c97b0 0x7fa7900c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39938/0 (socket says 172.21.15.32:39938) 2024-03-16T20:01:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.716+0000 7fa79481a640 1 -- 172.21.15.32:0/3076663378 learned_addr learned my addr 172.21.15.32:0/3076663378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:08.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.720+0000 7fa79481a640 1 -- 172.21.15.32:0/3076663378 >> v2:172.21.15.59:3300/0 conn(0x7fa790151ef0 msgr2=0x7fa7900c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:08.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.720+0000 7fa79481a640 1 --2- 172.21.15.32:0/3076663378 >> v2:172.21.15.59:3300/0 conn(0x7fa790151ef0 0x7fa7900c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.720+0000 7fa79481a640 1 -- 172.21.15.32:0/3076663378 >> v1:172.21.15.59:6789/0 conn(0x7fa7900c8720 legacy=0x7fa79015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.720+0000 7fa79481a640 1 -- 172.21.15.32:0/3076663378 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79015bfc0 con 0x7fa7900c97b0 2024-03-16T20:01:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.724+0000 7fa79481a640 1 --2- 172.21.15.32:0/3076663378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c97b0 0x7fa7900c81e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa7900cb3a0 tx=0x7fa78001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699821cf5f48a90a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa787fff640 1 -- 172.21.15.32:0/3076663378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800a89d0 con 0x7fa7900c97b0 2024-03-16T20:01:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa787fff640 1 -- 172.21.15.32:0/3076663378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7800a8b30 con 0x7fa7900c97b0 2024-03-16T20:01:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa787fff640 1 -- 172.21.15.32:0/3076663378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800b0440 con 0x7fa7900c97b0 2024-03-16T20:01:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 -- 172.21.15.32:0/3076663378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c97b0 msgr2=0x7fa7900c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 --2- 172.21.15.32:0/3076663378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c97b0 0x7fa7900c81e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa7900cb3a0 tx=0x7fa78001ecd0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 -- 172.21.15.32:0/3076663378 shutdown_connections 2024-03-16T20:01:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 --2- 172.21.15.32:0/3076663378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c97b0 0x7fa7900c81e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 --2- 172.21.15.32:0/3076663378 >> v2:172.21.15.59:3300/0 conn(0x7fa790151ef0 0x7fa7900c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 -- 172.21.15.32:0/3076663378 >> 172.21.15.32:0/3076663378 conn(0x7fa790070f90 msgr2=0x7fa7900713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 -- 172.21.15.32:0/3076663378 shutdown_connections 2024-03-16T20:01:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.728+0000 7fa79727f640 1 -- 172.21.15.32:0/3076663378 wait complete. 2024-03-16T20:01:08.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 Processor -- start 2024-03-16T20:01:08.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 -- start start 2024-03-16T20:01:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7900c97b0 0x7fa7900c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa790151ef0 0x7fa7900c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900b3650 con 0x7fa7900c8720 2024-03-16T20:01:08.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900c5180 con 0x7fa790151ef0 2024-03-16T20:01:08.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79727f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901f33c0 con 0x7fa7900c97b0 2024-03-16T20:01:08.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79501b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79581c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa790151ef0 0x7fa7900c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79501b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39940/0 (socket says 172.21.15.32:39940) 2024-03-16T20:01:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79501b640 1 -- 172.21.15.32:0/1043070121 learned_addr learned my addr 172.21.15.32:0/1043070121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.732+0000 7fa79481a640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7900c97b0 0x7fa7900c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.740+0000 7fa79501b640 1 -- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7900c97b0 msgr2=0x7fa7900c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.740+0000 7fa79501b640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7900c97b0 0x7fa7900c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.740+0000 7fa79501b640 1 -- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa790151ef0 msgr2=0x7fa7900c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.740+0000 7fa79501b640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa790151ef0 0x7fa7900c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.740+0000 7fa79501b640 1 -- 172.21.15.32:0/1043070121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800094d0 con 0x7fa7900c8720 2024-03-16T20:01:08.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.744+0000 7fa79501b640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e870 tx=0x7fa78c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c009510 con 0x7fa7900c8720 2024-03-16T20:01:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c00eec0 con 0x7fa7900c8720 2024-03-16T20:01:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00cb40 con 0x7fa7900c8720 2024-03-16T20:01:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa79727f640 1 -- 172.21.15.32:0/1043070121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7901f3f40 con 0x7fa7900c8720 2024-03-16T20:01:08.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa79727f640 1 -- 172.21.15.32:0/1043070121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7901f3610 con 0x7fa7900c8720 2024-03-16T20:01:08.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.748+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76c0052a0 con 0x7fa7900c8720 2024-03-16T20:01:08.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.784+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa78c00cca0 con 0x7fa7900c8720 2024-03-16T20:01:08.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.784+0000 7fa785ffb640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 0x7fa774069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:08.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.784+0000 7fa79481a640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 0x7fa774069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:08.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:08.784+0000 7fa79481a640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 0x7fa774069380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b2000 tx=0x7fa7800aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.060+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa78c055e80 con 0x7fa7900c8720 2024-03-16T20:01:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.060+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa78c052120 con 0x7fa7900c8720 2024-03-16T20:01:09.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.364+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa76c005900 con 0x7fa7900c8720 2024-03-16T20:01:09.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.460+0000 7fa785ffb640 1 -- 172.21.15.32:0/1043070121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fa76c005900 con 0x7fa7900c8720 2024-03-16T20:01:09.437 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:09.437 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:09.437 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:09.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 msgr2=0x7fa774069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 0x7fa774069380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b2000 tx=0x7fa7800aa070 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 msgr2=0x7fa79014b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e870 tx=0x7fa78c00ec70 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 shutdown_connections 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa774066ec0 0x7fa774069380 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa790151ef0 0x7fa7900c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7900c97b0 0x7fa7900c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 --2- 172.21.15.32:0/1043070121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7900c8720 0x7fa79014b3f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 >> 172.21.15.32:0/1043070121 conn(0x7fa790070f90 msgr2=0x7fa7900ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 shutdown_connections 2024-03-16T20:01:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:09.464+0000 7fa7737fe640 1 -- 172.21.15.32:0/1043070121 wait complete. 2024-03-16T20:01:10.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:10.059 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:11.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 Processor -- start 2024-03-16T20:01:11.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 -- start start 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc6040725e0 0x7fc604073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc60415a420 0x7fc60415a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc604151020 con 0x7fc604150c40 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60415a000 con 0x7fc6040725e0 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc60a785640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60415add0 con 0x7fc60415a420 2024-03-16T20:01:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc6037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc6040725e0 0x7fc604073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc6037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc6040725e0 0x7fc604073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35474/0 (socket says 172.21.15.32:35474) 2024-03-16T20:01:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc6037fe640 1 -- 172.21.15.32:0/1293941432 learned_addr learned my addr 172.21.15.32:0/1293941432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.196+0000 7fc608d22640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc60415a420 0x7fc60415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.200+0000 7fc6037fe640 1 -- 172.21.15.32:0/1293941432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc60415a420 msgr2=0x7fc60415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.200+0000 7fc6037fe640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc60415a420 0x7fc60415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.200+0000 7fc6037fe640 1 -- 172.21.15.32:0/1293941432 >> v1:172.21.15.59:6789/0 conn(0x7fc604150c40 legacy=0x7fc604070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.200+0000 7fc6037fe640 1 -- 172.21.15.32:0/1293941432 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604072da0 con 0x7fc6040725e0 2024-03-16T20:01:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.204+0000 7fc6037fe640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6040725e0 0x7fc604073070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009870 tx=0x7fc5f801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e8902cab027740 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:11.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.208+0000 7fc602ffd640 1 -- 172.21.15.32:0/1293941432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f80a9620 con 0x7fc6040725e0 2024-03-16T20:01:11.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.208+0000 7fc602ffd640 1 -- 172.21.15.32:0/1293941432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f80a9780 con 0x7fc6040725e0 2024-03-16T20:01:11.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.208+0000 7fc602ffd640 1 -- 172.21.15.32:0/1293941432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f80a9b10 con 0x7fc6040725e0 2024-03-16T20:01:11.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- 172.21.15.32:0/1293941432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6040725e0 msgr2=0x7fc604073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6040725e0 0x7fc604073070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009870 tx=0x7fc5f801c560 comp rx=0 tx=0).stop 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- 172.21.15.32:0/1293941432 shutdown_connections 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc60415a420 0x7fc60415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- 172.21.15.32:0/1293941432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6040725e0 0x7fc604073070 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- 172.21.15.32:0/1293941432 >> 172.21.15.32:0/1293941432 conn(0x7fc60406c0d0 msgr2=0x7fc60406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- 172.21.15.32:0/1293941432 shutdown_connections 2024-03-16T20:01:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- 172.21.15.32:0/1293941432 wait complete. 2024-03-16T20:01:11.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 Processor -- start 2024-03-16T20:01:11.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- start start 2024-03-16T20:01:11.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc6040725e0 0x7fc604145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc604150c40 0x7fc6041f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60414ba20 con 0x7fc60415a420 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604148510 con 0x7fc604150c40 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc60a785640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6041491e0 con 0x7fc6040725e0 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.212+0000 7fc608d22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.216+0000 7fc608d22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35488/0 (socket says 172.21.15.32:35488) 2024-03-16T20:01:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.216+0000 7fc608d22640 1 -- 172.21.15.32:0/154209636 learned_addr learned my addr 172.21.15.32:0/154209636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.216+0000 7fc603fff640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc6040725e0 0x7fc604145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.216+0000 7fc6037fe640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc604150c40 0x7fc6041f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 -- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc6040725e0 msgr2=0x7fc604145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc6040725e0 0x7fc604145220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 -- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc604150c40 msgr2=0x7fc6041f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc604150c40 0x7fc6041f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 -- 172.21.15.32:0/154209636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f80094d0 con 0x7fc60415a420 2024-03-16T20:01:11.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.220+0000 7fc608d22640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc007e70 tx=0x7fc5fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:11.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc014620 con 0x7fc60415a420 2024-03-16T20:01:11.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc0147a0 con 0x7fc60415a420 2024-03-16T20:01:11.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc604149a80 con 0x7fc60415a420 2024-03-16T20:01:11.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc014b50 con 0x7fc60415a420 2024-03-16T20:01:11.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc604148720 con 0x7fc60415a420 2024-03-16T20:01:11.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.228+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604068000 con 0x7fc60415a420 2024-03-16T20:01:11.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.512+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc5fc014cb0 con 0x7fc60415a420 2024-03-16T20:01:11.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.512+0000 7fc600ff9640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 0x7fc5f0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:11.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.512+0000 7fc603fff640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 0x7fc5f0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:11.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.512+0000 7fc603fff640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 0x7fc5f0069330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4007fd0 tx=0x7fc5f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:11.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.528+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc5fc056db0 con 0x7fc60415a420 2024-03-16T20:01:11.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.528+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc5fc053050 con 0x7fc60415a420 2024-03-16T20:01:11.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.800+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6040722b0 con 0x7fc60415a420 2024-03-16T20:01:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.836+0000 7fc600ff9640 1 -- 172.21.15.32:0/154209636 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc5fc056710 con 0x7fc60415a420 2024-03-16T20:01:11.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:11.812 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 msgr2=0x7fc5f0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 0x7fc5f0069330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4007fd0 tx=0x7fc5f4009040 comp rx=0 tx=0).stop 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 msgr2=0x7fc60415c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc007e70 tx=0x7fc5fc00e4b0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 shutdown_connections 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc5f0066e70 0x7fc5f0069330 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60415a420 0x7fc60415c210 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc604150c40 0x7fc6041f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 --2- 172.21.15.32:0/154209636 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc6040725e0 0x7fc604145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 >> 172.21.15.32:0/154209636 conn(0x7fc60406c0d0 msgr2=0x7fc60406f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 shutdown_connections 2024-03-16T20:01:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:11.840+0000 7fc60a785640 1 -- 172.21.15.32:0/154209636 wait complete. 2024-03-16T20:01:11.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:01:11.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:01:12.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:12.442 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:13.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:13.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 Processor -- start 2024-03-16T20:01:13.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 -- start start 2024-03-16T20:01:13.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df4154350 0x7f9df406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9df4072a10 0x7f9df4072e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df406bae0 con 0x7f9df40c2d70 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df415a540 con 0x7f9df4072a10 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.576+0000 7f9dfbfde640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40c7fd0 con 0x7f9df4154350 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9df4072a10 0x7f9df4072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9df4072a10 0x7f9df4072e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35502/0 (socket says 172.21.15.32:35502) 2024-03-16T20:01:13.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 -- 172.21.15.32:0/3808434837 learned_addr learned my addr 172.21.15.32:0/3808434837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:13.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9df9d7a640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df4154350 0x7f9df406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 -- 172.21.15.32:0/3808434837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df4154350 msgr2=0x7f9df406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df4154350 0x7f9df406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 -- 172.21.15.32:0/3808434837 >> v1:172.21.15.59:6789/0 conn(0x7f9df40c2d70 legacy=0x7f9df415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.580+0000 7f9dfa57b640 1 -- 172.21.15.32:0/3808434837 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df406bd10 con 0x7f9df4072a10 2024-03-16T20:01:13.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.584+0000 7f9dfa57b640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df4072e60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009e50 tx=0x7f9dec01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3192fbc48e3f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9df8d78640 1 -- 172.21.15.32:0/3808434837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0a5660 con 0x7f9df4072a10 2024-03-16T20:01:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9df8d78640 1 -- 172.21.15.32:0/3808434837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec0a57c0 con 0x7f9df4072a10 2024-03-16T20:01:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9df8d78640 1 -- 172.21.15.32:0/3808434837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0a5b50 con 0x7f9df4072a10 2024-03-16T20:01:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 -- 172.21.15.32:0/3808434837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 msgr2=0x7f9df4072e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df4072e60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009e50 tx=0x7f9dec01c9a0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 -- 172.21.15.32:0/3808434837 shutdown_connections 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df4072e60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/3808434837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df4154350 0x7f9df406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 -- 172.21.15.32:0/3808434837 >> 172.21.15.32:0/3808434837 conn(0x7f9df406afc0 msgr2=0x7f9df406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 -- 172.21.15.32:0/3808434837 shutdown_connections 2024-03-16T20:01:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.592+0000 7f9dfbfde640 1 -- 172.21.15.32:0/3808434837 wait complete. 2024-03-16T20:01:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 Processor -- start 2024-03-16T20:01:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 -- start start 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df40c2d70 0x7f9df40ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df4154350 0x7f9df41f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df415bb70 con 0x7f9df4072a10 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40cbaf0 con 0x7f9df4154350 2024-03-16T20:01:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfbfde640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4158490 con 0x7f9df40c2d70 2024-03-16T20:01:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9df9d7a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9dfa57b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df4154350 0x7f9df41f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9df9d7a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35508/0 (socket says 172.21.15.32:35508) 2024-03-16T20:01:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9df9d7a640 1 -- 172.21.15.32:0/6249150 learned_addr learned my addr 172.21.15.32:0/6249150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.596+0000 7f9df9579640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df40c2d70 0x7f9df40ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.604+0000 7f9df9d7a640 1 -- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df40c2d70 msgr2=0x7f9df40ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:13.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.604+0000 7f9df9d7a640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df40c2d70 0x7f9df40ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.604+0000 7f9df9d7a640 1 -- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df4154350 msgr2=0x7f9df41f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:13.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.604+0000 7f9df9d7a640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df4154350 0x7f9df41f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:13.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.604+0000 7f9df9d7a640 1 -- 172.21.15.32:0/6249150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec009b30 con 0x7f9df4072a10 2024-03-16T20:01:13.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.608+0000 7f9df9d7a640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e870 tx=0x7f9df000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:13.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0009510 con 0x7f9df4072a10 2024-03-16T20:01:13.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df000eec0 con 0x7f9df4072a10 2024-03-16T20:01:13.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df415d0b0 con 0x7f9df4072a10 2024-03-16T20:01:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df000cb40 con 0x7f9df4072a10 2024-03-16T20:01:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df40cb150 con 0x7f9df4072a10 2024-03-16T20:01:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.612+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df40674e0 con 0x7f9df4072a10 2024-03-16T20:01:13.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.648+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9df000cca0 con 0x7f9df4072a10 2024-03-16T20:01:13.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.652+0000 7f9dea7fc640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 0x7f9dd8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:13.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.652+0000 7f9df9579640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 0x7f9dd8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:13.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.652+0000 7f9df9579640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 0x7f9dd8069350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9df414a8b0 tx=0x7f9de4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.944+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9df0055ee0 con 0x7f9df4072a10 2024-03-16T20:01:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:13.944+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9df0052180 con 0x7f9df4072a10 2024-03-16T20:01:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.180+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df4068000 con 0x7f9df4072a10 2024-03-16T20:01:14.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dea7fc640 1 -- 172.21.15.32:0/6249150 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9df0021020 con 0x7f9df4072a10 2024-03-16T20:01:14.178 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:14.178 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:14.179 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:14.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 msgr2=0x7f9dd8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:14.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 0x7f9dd8069350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9df414a8b0 tx=0x7f9de4009040 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 msgr2=0x7f9df415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e870 tx=0x7f9df000ec70 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 shutdown_connections 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9dd8066e90 0x7f9dd8069350 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df4154350 0x7f9df41f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9df40c2d70 0x7f9df40ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 --2- 172.21.15.32:0/6249150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df4072a10 0x7f9df415c700 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 >> 172.21.15.32:0/6249150 conn(0x7f9df406afc0 msgr2=0x7f9df4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 shutdown_connections 2024-03-16T20:01:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:14.204+0000 7f9dfbfde640 1 -- 172.21.15.32:0/6249150 wait complete. 2024-03-16T20:01:14.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:14.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:15.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 Processor -- start 2024-03-16T20:01:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 -- start start 2024-03-16T20:01:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf80c6fd0 0x7f1cf815a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cf80c8080 0x7f1cf80c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf8072850 con 0x7f1cf8150de0 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80c8e50 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cfc904640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80c7910 con 0x7f1cf80c8080 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cf5d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf80c6fd0 0x7f1cf815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.936+0000 7f1cf6d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cf80c8080 0x7f1cf80c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf80c6fd0 0x7f1cf815a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35510/0 (socket says 172.21.15.32:35510) 2024-03-16T20:01:15.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 -- 172.21.15.32:0/3338353388 learned_addr learned my addr 172.21.15.32:0/3338353388 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:15.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 -- 172.21.15.32:0/3338353388 >> v2:172.21.15.59:3300/0 conn(0x7f1cf80c8080 msgr2=0x7f1cf80c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:15.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 --2- 172.21.15.32:0/3338353388 >> v2:172.21.15.59:3300/0 conn(0x7f1cf80c8080 0x7f1cf80c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:15.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 -- 172.21.15.32:0/3338353388 >> v1:172.21.15.59:6789/0 conn(0x7f1cf8150de0 legacy=0x7f1cf80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:15.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.940+0000 7f1cf5d9b640 1 -- 172.21.15.32:0/3338353388 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80c8510 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.944+0000 7f1cf5d9b640 1 --2- 172.21.15.32:0/3338353388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf815a140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80c6c60 tx=0x7f1cec01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7e1c009ac9098e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cf559a640 1 -- 172.21.15.32:0/3338353388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0ad660 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cf559a640 1 -- 172.21.15.32:0/3338353388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec0ad7c0 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cf559a640 1 -- 172.21.15.32:0/3338353388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0adb50 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 -- 172.21.15.32:0/3338353388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 msgr2=0x7f1cf815a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 --2- 172.21.15.32:0/3338353388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf815a140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80c6c60 tx=0x7f1cec01c450 comp rx=0 tx=0).stop 2024-03-16T20:01:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 -- 172.21.15.32:0/3338353388 shutdown_connections 2024-03-16T20:01:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 --2- 172.21.15.32:0/3338353388 >> v2:172.21.15.59:3300/0 conn(0x7f1cf80c8080 0x7f1cf80c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 --2- 172.21.15.32:0/3338353388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf815a140 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 -- 172.21.15.32:0/3338353388 >> 172.21.15.32:0/3338353388 conn(0x7f1cf80cc7a0 msgr2=0x7f1cf80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 -- 172.21.15.32:0/3338353388 shutdown_connections 2024-03-16T20:01:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.952+0000 7f1cfc904640 1 -- 172.21.15.32:0/3338353388 wait complete. 2024-03-16T20:01:15.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 Processor -- start 2024-03-16T20:01:15.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 -- start start 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf8160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 0x7f1cf806b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf8150de0 0x7f1cf81f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf815fd30 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf814ba00 con 0x7f1cf8150de0 2024-03-16T20:01:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cfc904640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf81f2dc0 con 0x7f1cf80c8080 2024-03-16T20:01:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cf5d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 0x7f1cf806b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cf6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf8150de0 0x7f1cf81f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cf5d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 0x7f1cf806b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:35554/0 (socket says 172.21.15.32:35554) 2024-03-16T20:01:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cf5d9b640 1 -- 172.21.15.32:0/538880136 learned_addr learned my addr 172.21.15.32:0/538880136 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.956+0000 7f1cf659c640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf8160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.960+0000 7f1cf659c640 1 -- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 msgr2=0x7f1cf806b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.960+0000 7f1cf659c640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 0x7f1cf806b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.960+0000 7f1cf659c640 1 -- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf8150de0 msgr2=0x7f1cf81f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.960+0000 7f1cf659c640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf8150de0 0x7f1cf81f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.960+0000 7f1cf659c640 1 -- 172.21.15.32:0/538880136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094d0 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.964+0000 7f1cf659c640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf8160950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800c390 tx=0x7f1ce800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.968+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8003600 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.968+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce8003760 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.968+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8003b10 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.968+0000 7f1cfc904640 1 -- 172.21.15.32:0/538880136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf815fa90 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.968+0000 7f1cfc904640 1 -- 172.21.15.32:0/538880136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf814b720 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.972+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd00052a0 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.996+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1ce8003c70 con 0x7f1cf80c6fd0 2024-03-16T20:01:15.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.996+0000 7f1ce6ffd640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:15.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.996+0000 7f1cf5d9b640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:15.996+0000 7f1cf5d9b640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 0x7f1cd8069330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1cec01c490 tx=0x7f1cec0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.300+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1ce8054c40 con 0x7f1cf80c6fd0 2024-03-16T20:01:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.300+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1ce8050ee0 con 0x7f1cf80c6fd0 2024-03-16T20:01:16.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.496+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd0005900 con 0x7f1cf80c6fd0 2024-03-16T20:01:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.516+0000 7f1ce6ffd640 1 -- 172.21.15.32:0/538880136 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1cd0005900 con 0x7f1cf80c6fd0 2024-03-16T20:01:16.490 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:16.490 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 msgr2=0x7f1cd8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 0x7f1cd8069330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1cec01c490 tx=0x7f1cec0a7000 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 msgr2=0x7f1cf8160950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf8160950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800c390 tx=0x7f1ce800c850 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 shutdown_connections 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf8150de0 0x7f1cf81f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cf80c8080 0x7f1cf806b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 --2- 172.21.15.32:0/538880136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf80c6fd0 0x7f1cf8160950 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 >> 172.21.15.32:0/538880136 conn(0x7f1cf80cc7a0 msgr2=0x7f1cf80c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 shutdown_connections 2024-03-16T20:01:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:16.520+0000 7f1ce4ff9640 1 -- 172.21.15.32:0/538880136 wait complete. 2024-03-16T20:01:17.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:17.119 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:18.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 Processor -- start 2024-03-16T20:01:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 -- start start 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1724151fb0 0x7f172406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1724066cd0 0x7f17241625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1724157d40 con 0x7f1724067140 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17240c6c60 con 0x7f1724151fb0 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.252+0000 7f1729d41640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17240c6f40 con 0x7f1724066cd0 2024-03-16T20:01:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.256+0000 7f17237fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1724151fb0 0x7f172406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.256+0000 7f17237fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1724151fb0 0x7f172406afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35526/0 (socket says 172.21.15.32:35526) 2024-03-16T20:01:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.256+0000 7f17237fe640 1 -- 172.21.15.32:0/3297766415 learned_addr learned my addr 172.21.15.32:0/3297766415 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.256+0000 7f1723fff640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1724066cd0 0x7f17241625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.260+0000 7f17237fe640 1 -- 172.21.15.32:0/3297766415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1724066cd0 msgr2=0x7f17241625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.260+0000 7f17237fe640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1724066cd0 0x7f17241625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.260+0000 7f17237fe640 1 -- 172.21.15.32:0/3297766415 >> v1:172.21.15.59:6789/0 conn(0x7f1724067140 legacy=0x7f1724072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.260+0000 7f17237fe640 1 -- 172.21.15.32:0/3297766415 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1724158000 con 0x7f1724151fb0 2024-03-16T20:01:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.260+0000 7f17237fe640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724151fb0 0x7f172406afc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1714009870 tx=0x7f171401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236729d4063a3ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:18.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f17227fc640 1 -- 172.21.15.32:0/3297766415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140a8a50 con 0x7f1724151fb0 2024-03-16T20:01:18.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f17227fc640 1 -- 172.21.15.32:0/3297766415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17140a8bb0 con 0x7f1724151fb0 2024-03-16T20:01:18.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f17227fc640 1 -- 172.21.15.32:0/3297766415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140b0440 con 0x7f1724151fb0 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- 172.21.15.32:0/3297766415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724151fb0 msgr2=0x7f172406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724151fb0 0x7f172406afc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1714009870 tx=0x7f171401ed50 comp rx=0 tx=0).stop 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- 172.21.15.32:0/3297766415 shutdown_connections 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1724066cd0 0x7f17241625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 --2- 172.21.15.32:0/3297766415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724151fb0 0x7f172406afc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- 172.21.15.32:0/3297766415 >> 172.21.15.32:0/3297766415 conn(0x7f172406f560 msgr2=0x7f172406f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- 172.21.15.32:0/3297766415 shutdown_connections 2024-03-16T20:01:18.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- 172.21.15.32:0/3297766415 wait complete. 2024-03-16T20:01:18.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 Processor -- start 2024-03-16T20:01:18.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.268+0000 7f1729d41640 1 -- start start 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1724151fb0 0x7f1724156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f172414a810 0x7f1724156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17241574e0 con 0x7f1724066cd0 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240c51d0 con 0x7f1724151fb0 2024-03-16T20:01:18.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1729d41640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1724155bb0 con 0x7f172414a810 2024-03-16T20:01:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f17237fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f17237fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35532/0 (socket says 172.21.15.32:35532) 2024-03-16T20:01:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f17237fe640 1 -- 172.21.15.32:0/517108482 learned_addr learned my addr 172.21.15.32:0/517108482 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1722ffd640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1724151fb0 0x7f1724156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.272+0000 7f1723fff640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f172414a810 0x7f1724156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.276+0000 7f17237fe640 1 -- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f172414a810 msgr2=0x7f1724156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.276+0000 7f17237fe640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f172414a810 0x7f1724156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.276+0000 7f17237fe640 1 -- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1724151fb0 msgr2=0x7f1724156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.276+0000 7f17237fe640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1724151fb0 0x7f1724156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.276+0000 7f17237fe640 1 -- 172.21.15.32:0/517108482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140094d0 con 0x7f1724066cd0 2024-03-16T20:01:18.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.280+0000 7f17237fe640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f171401b960 tx=0x7f171401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140b1ce0 con 0x7f1724066cd0 2024-03-16T20:01:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17140b1e40 con 0x7f1724066cd0 2024-03-16T20:01:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f1729d41640 1 -- 172.21.15.32:0/517108482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17240c4c10 con 0x7f1724066cd0 2024-03-16T20:01:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140b9680 con 0x7f1724066cd0 2024-03-16T20:01:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f1729d41640 1 -- 172.21.15.32:0/517108482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17241576f0 con 0x7f1724066cd0 2024-03-16T20:01:18.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.284+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc0052a0 con 0x7f1724066cd0 2024-03-16T20:01:18.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.320+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f17140b7070 con 0x7f1724066cd0 2024-03-16T20:01:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.320+0000 7f170bfff640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 0x7f1710069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:18.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.320+0000 7f1722ffd640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 0x7f1710069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.320+0000 7f1722ffd640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 0x7f1710069300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f172414aee0 tx=0x7f1718009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:18.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.324+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f17140f3890 con 0x7f1724066cd0 2024-03-16T20:01:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.608+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f17140efb30 con 0x7f1724066cd0 2024-03-16T20:01:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.868+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16fc005900 con 0x7f1724066cd0 2024-03-16T20:01:18.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f170bfff640 1 -- 172.21.15.32:0/517108482 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f16fc005900 con 0x7f1724066cd0 2024-03-16T20:01:18.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:18.946 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:18.946 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 msgr2=0x7f1710069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 0x7f1710069300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f172414aee0 tx=0x7f1718009040 comp rx=0 tx=0).stop 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 msgr2=0x7f172414a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f171401b960 tx=0x7f171401baf0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 shutdown_connections 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1710066e40 0x7f1710069300 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f172414a810 0x7f1724156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1724151fb0 0x7f1724156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 --2- 172.21.15.32:0/517108482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1724066cd0 0x7f172414a2d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:18.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 >> 172.21.15.32:0/517108482 conn(0x7f172406f560 msgr2=0x7f1724073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:18.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.972+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 shutdown_connections 2024-03-16T20:01:18.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:18.976+0000 7f1709ffb640 1 -- 172.21.15.32:0/517108482 wait complete. 2024-03-16T20:01:19.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:19.572 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:20.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:20.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.704+0000 7f447636f640 1 Processor -- start 2024-03-16T20:01:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.704+0000 7f447636f640 1 -- start start 2024-03-16T20:01:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447636f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4470151000 0x7f44700c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447636f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44700c8980 0x7f447015a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447636f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44700c8ec0 con 0x7f44700c36b0 2024-03-16T20:01:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447636f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4470072f10 con 0x7f44700c8980 2024-03-16T20:01:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447636f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44700c5000 con 0x7f4470151000 2024-03-16T20:01:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447490c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44700c8980 0x7f447015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447490c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44700c8980 0x7f447015a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32896/0 (socket says 172.21.15.32:32896) 2024-03-16T20:01:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f447490c640 1 -- 172.21.15.32:0/4086945024 learned_addr learned my addr 172.21.15.32:0/4086945024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.708+0000 7f446ffff640 1 --2- 172.21.15.32:0/4086945024 >> v2:172.21.15.59:3300/0 conn(0x7f4470151000 0x7f44700c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.712+0000 7f447490c640 1 -- 172.21.15.32:0/4086945024 >> v2:172.21.15.59:3300/0 conn(0x7f4470151000 msgr2=0x7f44700c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.712+0000 7f447490c640 1 --2- 172.21.15.32:0/4086945024 >> v2:172.21.15.59:3300/0 conn(0x7f4470151000 0x7f44700c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.712+0000 7f447490c640 1 -- 172.21.15.32:0/4086945024 >> v1:172.21.15.32:6789/0 conn(0x7f44700c36b0 legacy=0x7f44700c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.712+0000 7f447490c640 1 -- 172.21.15.32:0/4086945024 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470072950 con 0x7f44700c8980 2024-03-16T20:01:20.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.720+0000 7f447490c640 1 --2- 172.21.15.32:0/4086945024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c8980 0x7f447015a240 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f44700725d0 tx=0x7f446801ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73e89cad8992cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:20.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f446effd640 1 -- 172.21.15.32:0/4086945024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44680a79d0 con 0x7f44700c8980 2024-03-16T20:01:20.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f446effd640 1 -- 172.21.15.32:0/4086945024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44680a7b30 con 0x7f44700c8980 2024-03-16T20:01:20.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f446effd640 1 -- 172.21.15.32:0/4086945024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44680af3f0 con 0x7f44700c8980 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 -- 172.21.15.32:0/4086945024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c8980 msgr2=0x7f447015a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 --2- 172.21.15.32:0/4086945024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c8980 0x7f447015a240 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f44700725d0 tx=0x7f446801ec00 comp rx=0 tx=0).stop 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 -- 172.21.15.32:0/4086945024 shutdown_connections 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 --2- 172.21.15.32:0/4086945024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c8980 0x7f447015a240 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 --2- 172.21.15.32:0/4086945024 >> v2:172.21.15.59:3300/0 conn(0x7f4470151000 0x7f44700c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 -- 172.21.15.32:0/4086945024 >> 172.21.15.32:0/4086945024 conn(0x7f447006c0d0 msgr2=0x7f447006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 -- 172.21.15.32:0/4086945024 shutdown_connections 2024-03-16T20:01:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.724+0000 7f447636f640 1 -- 172.21.15.32:0/4086945024 wait complete. 2024-03-16T20:01:20.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 Processor -- start 2024-03-16T20:01:20.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 -- start start 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44700c8980 0x7f447014f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4470151000 0x7f447014fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447015a090 con 0x7f44700c36b0 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470143e50 con 0x7f4470151000 2024-03-16T20:01:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447636f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4470158be0 con 0x7f44700c8980 2024-03-16T20:01:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f446ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f447490c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4470151000 0x7f447014fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f446ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32910/0 (socket says 172.21.15.32:32910) 2024-03-16T20:01:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f446ffff640 1 -- 172.21.15.32:0/3356995082 learned_addr learned my addr 172.21.15.32:0/3356995082 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.728+0000 7f446f7fe640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44700c8980 0x7f447014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.732+0000 7f446ffff640 1 -- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44700c8980 msgr2=0x7f447014f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.732+0000 7f446ffff640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44700c8980 0x7f447014f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.732+0000 7f446ffff640 1 -- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4470151000 msgr2=0x7f447014fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.732+0000 7f446ffff640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4470151000 0x7f447014fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:20.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.732+0000 7f446ffff640 1 -- 172.21.15.32:0/3356995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468009b30 con 0x7f44700c36b0 2024-03-16T20:01:20.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.736+0000 7f446ffff640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f446000e870 tx=0x7f446000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:20.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460009510 con 0x7f44700c36b0 2024-03-16T20:01:20.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447014eb80 con 0x7f44700c36b0 2024-03-16T20:01:20.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470159db0 con 0x7f44700c36b0 2024-03-16T20:01:20.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470068000 con 0x7f44700c36b0 2024-03-16T20:01:20.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446000eec0 con 0x7f44700c36b0 2024-03-16T20:01:20.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:20.740+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460005bc0 con 0x7f44700c36b0 2024-03-16T20:01:21.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.028+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f446000c940 con 0x7f44700c36b0 2024-03-16T20:01:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.028+0000 7f446cff9640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 0x7f445c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.028+0000 7f446f7fe640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 0x7f445c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.028+0000 7f446f7fe640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 0x7f445c069300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f44701612c0 tx=0x7f4464009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.052+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4460055ec0 con 0x7f44700c36b0 2024-03-16T20:01:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.052+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4460052160 con 0x7f44700c36b0 2024-03-16T20:01:21.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.260+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44701478c0 con 0x7f44700c36b0 2024-03-16T20:01:21.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f446cff9640 1 -- 172.21.15.32:0/3356995082 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4460021070 con 0x7f44700c36b0 2024-03-16T20:01:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:21.250 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:21.250 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 msgr2=0x7f445c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 0x7f445c069300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f44701612c0 tx=0x7f4464009040 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 msgr2=0x7f4470143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f446000e870 tx=0x7f446000ec70 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 shutdown_connections 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f445c066e40 0x7f445c069300 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4470151000 0x7f447014fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44700c8980 0x7f447014f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 --2- 172.21.15.32:0/3356995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44700c36b0 0x7f4470143910 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 >> 172.21.15.32:0/3356995082 conn(0x7f447006c0d0 msgr2=0x7f4470070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 shutdown_connections 2024-03-16T20:01:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:21.276+0000 7f447636f640 1 -- 172.21.15.32:0/3356995082 wait complete. 2024-03-16T20:01:21.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:21.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:22.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 Processor -- start 2024-03-16T20:01:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 -- start start 2024-03-16T20:01:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faa200c6420 0x7faa200c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa20070760 con 0x7faa200c6040 2024-03-16T20:01:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa200731c0 con 0x7faa20158390 2024-03-16T20:01:22.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.012+0000 7faa2829e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa200734a0 con 0x7faa200c6420 2024-03-16T20:01:22.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.016+0000 7faa25839640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faa200c6420 0x7faa200c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:22.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.016+0000 7faa25839640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faa200c6420 0x7faa200c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32914/0 (socket says 172.21.15.32:32914) 2024-03-16T20:01:22.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.016+0000 7faa25839640 1 -- 172.21.15.32:0/3211610166 learned_addr learned my addr 172.21.15.32:0/3211610166 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:22.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.020+0000 7faa25038640 1 -- 172.21.15.32:0/3211610166 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569118093 0 0) 0x7faa200731c0 con 0x7faa20158390 2024-03-16T20:01:22.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.020+0000 7faa25038640 1 -- 172.21.15.32:0/3211610166 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa0c003530 con 0x7faa20158390 2024-03-16T20:01:22.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.020+0000 7faa25839640 1 -- 172.21.15.32:0/3211610166 >> v1:172.21.15.59:6789/0 conn(0x7faa20158390 legacy=0x7faa201516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:22.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.020+0000 7faa25839640 1 -- 172.21.15.32:0/3211610166 >> v1:172.21.15.32:6789/0 conn(0x7faa200c6040 legacy=0x7faa20157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:22.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.020+0000 7faa25839640 1 -- 172.21.15.32:0/3211610166 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2006fe30 con 0x7faa200c6420 2024-03-16T20:01:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.028+0000 7faa25839640 1 --2- 172.21.15.32:0/3211610166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa200c6420 0x7faa200c8800 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faa10009870 tx=0x7faa1001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33624a6b267efbef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa25038640 1 -- 172.21.15.32:0/3211610166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100a95f0 con 0x7faa200c6420 2024-03-16T20:01:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa25038640 1 -- 172.21.15.32:0/3211610166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa100a9750 con 0x7faa200c6420 2024-03-16T20:01:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa25038640 1 -- 172.21.15.32:0/3211610166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100a9ae0 con 0x7faa200c6420 2024-03-16T20:01:23.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 -- 172.21.15.32:0/3211610166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa200c6420 msgr2=0x7faa200c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:23.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 --2- 172.21.15.32:0/3211610166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa200c6420 0x7faa200c8800 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faa10009870 tx=0x7faa1001c530 comp rx=0 tx=0).stop 2024-03-16T20:01:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 -- 172.21.15.32:0/3211610166 shutdown_connections 2024-03-16T20:01:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 --2- 172.21.15.32:0/3211610166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa200c6420 0x7faa200c8800 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 -- 172.21.15.32:0/3211610166 >> 172.21.15.32:0/3211610166 conn(0x7faa2006c0d0 msgr2=0x7faa2006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 -- 172.21.15.32:0/3211610166 shutdown_connections 2024-03-16T20:01:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.032+0000 7faa2829e640 1 -- 172.21.15.32:0/3211610166 wait complete. 2024-03-16T20:01:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 Processor -- start 2024-03-16T20:01:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 -- start start 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa200c6040 0x7faa201462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa200c6420 0x7faa20152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20149d30 con 0x7faa20158390 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20154910 con 0x7faa200c6420 2024-03-16T20:01:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2829e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa20154da0 con 0x7faa200c6040 2024-03-16T20:01:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2683b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2603a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa200c6040 0x7faa201462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2683b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32926/0 (socket says 172.21.15.32:32926) 2024-03-16T20:01:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa2683b640 1 -- 172.21.15.32:0/774596861 learned_addr learned my addr 172.21.15.32:0/774596861 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.036+0000 7faa25839640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa200c6420 0x7faa20152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 -- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa200c6040 msgr2=0x7faa201462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa200c6040 0x7faa201462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 -- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa200c6420 msgr2=0x7faa20152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa200c6420 0x7faa20152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 -- 172.21.15.32:0/774596861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa100094d0 con 0x7faa20158390 2024-03-16T20:01:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.044+0000 7faa2683b640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c7c0 tx=0x7faa1800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18003b80 con 0x7faa20158390 2024-03-16T20:01:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18003ce0 con 0x7faa20158390 2024-03-16T20:01:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa20155450 con 0x7faa20158390 2024-03-16T20:01:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1800aa50 con 0x7faa20158390 2024-03-16T20:01:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa20149a50 con 0x7faa20158390 2024-03-16T20:01:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.052+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20068000 con 0x7faa20158390 2024-03-16T20:01:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.084+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faa1800abb0 con 0x7faa20158390 2024-03-16T20:01:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.084+0000 7faa16ffd640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 0x7faa04069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.084+0000 7faa2603a640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 0x7faa04069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.084+0000 7faa2603a640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 0x7faa04069350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faa1c007fd0 tx=0x7faa1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.088+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7faa18055f40 con 0x7faa20158390 2024-03-16T20:01:23.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.396+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7faa180521e0 con 0x7faa20158390 2024-03-16T20:01:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.608+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa20154a50 con 0x7faa20158390 2024-03-16T20:01:23.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.688+0000 7faa16ffd640 1 -- 172.21.15.32:0/774596861 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7faa1800ae60 con 0x7faa20158390 2024-03-16T20:01:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:23.664 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:23.664 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 msgr2=0x7faa04069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 0x7faa04069350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faa1c007fd0 tx=0x7faa1c009040 comp rx=0 tx=0).stop 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 msgr2=0x7faa20146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c7c0 tx=0x7faa1800cc80 comp rx=0 tx=0).stop 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 shutdown_connections 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa04066e90 0x7faa04069350 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa20158390 0x7faa20146d90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa200c6420 0x7faa20152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 --2- 172.21.15.32:0/774596861 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa200c6040 0x7faa201462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 >> 172.21.15.32:0/774596861 conn(0x7faa2006c0d0 msgr2=0x7faa2006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 shutdown_connections 2024-03-16T20:01:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:23.692+0000 7faa2829e640 1 -- 172.21.15.32:0/774596861 wait complete. 2024-03-16T20:01:24.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:24.289 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:25.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:25.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 Processor -- start 2024-03-16T20:01:25.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 -- start start 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba00154350 0x7fba0006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fba000c2d70 0x7fba0015a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0006bae0 con 0x7fba00072a10 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0015a540 con 0x7fba00154350 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fba06118640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba000c7fd0 con 0x7fba000c2d70 2024-03-16T20:01:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.424+0000 7fb9ff7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba00154350 0x7fba0006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:25.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba00154350 0x7fba0006c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32940/0 (socket says 172.21.15.32:32940) 2024-03-16T20:01:25.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/2702555446 learned_addr learned my addr 172.21.15.32:0/2702555446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:25.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/2702555446 >> v2:172.21.15.59:3300/0 conn(0x7fba000c2d70 msgr2=0x7fba0015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 --2- 172.21.15.32:0/2702555446 >> v2:172.21.15.59:3300/0 conn(0x7fba000c2d70 0x7fba0015a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/2702555446 >> v1:172.21.15.59:6789/0 conn(0x7fba00072a10 legacy=0x7fba000c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.428+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/2702555446 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0006bd10 con 0x7fba00154350 2024-03-16T20:01:25.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.436+0000 7fb9ff7fe640 1 --2- 172.21.15.32:0/2702555446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00154350 0x7fba0006c250 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fba000c1c10 tx=0x7fb9f001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc80eea7b79c344 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fb9fe7fc640 1 -- 172.21.15.32:0/2702555446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00ab600 con 0x7fba00154350 2024-03-16T20:01:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fb9fe7fc640 1 -- 172.21.15.32:0/2702555446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f00a7050 con 0x7fba00154350 2024-03-16T20:01:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fb9fe7fc640 1 -- 172.21.15.32:0/2702555446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00ab980 con 0x7fba00154350 2024-03-16T20:01:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 -- 172.21.15.32:0/2702555446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00154350 msgr2=0x7fba0006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 --2- 172.21.15.32:0/2702555446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00154350 0x7fba0006c250 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fba000c1c10 tx=0x7fb9f001c450 comp rx=0 tx=0).stop 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 -- 172.21.15.32:0/2702555446 shutdown_connections 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 --2- 172.21.15.32:0/2702555446 >> v2:172.21.15.59:3300/0 conn(0x7fba000c2d70 0x7fba0015a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 --2- 172.21.15.32:0/2702555446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00154350 0x7fba0006c250 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 -- 172.21.15.32:0/2702555446 >> 172.21.15.32:0/2702555446 conn(0x7fba0006afc0 msgr2=0x7fba0006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 -- 172.21.15.32:0/2702555446 shutdown_connections 2024-03-16T20:01:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.440+0000 7fba06118640 1 -- 172.21.15.32:0/2702555446 wait complete. 2024-03-16T20:01:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 Processor -- start 2024-03-16T20:01:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 -- start start 2024-03-16T20:01:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba000c2d70 0x7fba0015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba00154350 0x7fba0015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba001f27a0 con 0x7fba00072a10 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0006c630 con 0x7fba00154350 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fba06118640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba001f2990 con 0x7fba000c2d70 2024-03-16T20:01:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fb9ff7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:25.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fb9ff7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32944/0 (socket says 172.21.15.32:32944) 2024-03-16T20:01:25.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/457657066 learned_addr learned my addr 172.21.15.32:0/457657066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:25.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fb9feffd640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba000c2d70 0x7fba0015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:25.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.444+0000 7fb9fffff640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba00154350 0x7fba0015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.448+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba000c2d70 msgr2=0x7fba0015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.448+0000 7fb9ff7fe640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba000c2d70 0x7fba0015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.448+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba00154350 msgr2=0x7fba0015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.448+0000 7fb9ff7fe640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba00154350 0x7fba0015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.448+0000 7fb9ff7fe640 1 -- 172.21.15.32:0/457657066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00094d0 con 0x7fba00072a10 2024-03-16T20:01:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.452+0000 7fb9ff7fe640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00ae040 tx=0x7fb9f001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00b6620 con 0x7fba00072a10 2024-03-16T20:01:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f00b67a0 con 0x7fba00072a10 2024-03-16T20:01:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba001f2c50 con 0x7fba00072a10 2024-03-16T20:01:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00b6b50 con 0x7fba00072a10 2024-03-16T20:01:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba001f3210 con 0x7fba00072a10 2024-03-16T20:01:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.456+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba000674e0 con 0x7fba00072a10 2024-03-16T20:01:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.492+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb9f00b2040 con 0x7fba00072a10 2024-03-16T20:01:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.496+0000 7fb9e7fff640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 0x7fb9ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:25.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.496+0000 7fb9feffd640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 0x7fb9ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:25.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.496+0000 7fb9feffd640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 0x7fb9ec069300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fba00157b30 tx=0x7fb9f4009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.800+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb9f00f0440 con 0x7fba00072a10 2024-03-16T20:01:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:25.800+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb9f00ec6e0 con 0x7fba00072a10 2024-03-16T20:01:26.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.080+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba0015c090 con 0x7fba00072a10 2024-03-16T20:01:26.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.100+0000 7fb9e7fff640 1 -- 172.21.15.32:0/457657066 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb9f00b0070 con 0x7fba00072a10 2024-03-16T20:01:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:26.077 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:26.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 msgr2=0x7fb9ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 0x7fb9ec069300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fba00157b30 tx=0x7fb9f4009110 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 msgr2=0x7fba00153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00ae040 tx=0x7fb9f001ac90 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 shutdown_connections 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb9ec066e40 0x7fb9ec069300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba00154350 0x7fba0015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba000c2d70 0x7fba0015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 --2- 172.21.15.32:0/457657066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00072a10 0x7fba00153870 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 >> 172.21.15.32:0/457657066 conn(0x7fba0006afc0 msgr2=0x7fba0006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 shutdown_connections 2024-03-16T20:01:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:26.104+0000 7fba06118640 1 -- 172.21.15.32:0/457657066 wait complete. 2024-03-16T20:01:26.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:26.701 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:27.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 Processor -- start 2024-03-16T20:01:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 -- start start 2024-03-16T20:01:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60e4073010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e41546a0 con 0x7f60e41542c0 2024-03-16T20:01:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e4073550 con 0x7f60e415dad0 2024-03-16T20:01:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60eb635640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4159f20 con 0x7f60e4070360 2024-03-16T20:01:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60e93d1640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60e41542c0 0x7f60e4072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55482/0 (socket says 172.21.15.32:55482) 2024-03-16T20:01:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60e93d1640 1 -- 172.21.15.32:0/796826415 learned_addr learned my addr 172.21.15.32:0/796826415 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.836+0000 7f60e8bd0640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60e4073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.840+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135739131 0 0) 0x7f60e41546a0 con 0x7f60e41542c0 2024-03-16T20:01:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.840+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d0003530 con 0x7f60e41542c0 2024-03-16T20:01:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153007792 0 0) 0x7f60e4073550 con 0x7f60e415dad0 2024-03-16T20:01:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e41546a0 con 0x7f60e415dad0 2024-03-16T20:01:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552899613 0 0) 0x7f60d0003530 con 0x7f60e41542c0 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 msgr2=0x7f60e4073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60e4073010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 >> v1:172.21.15.59:6789/0 conn(0x7f60e415dad0 legacy=0x7f60e41648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e415a1e0 con 0x7f60e41542c0 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f60e0001680 con 0x7f60e41542c0 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 >> v1:172.21.15.32:6789/0 conn(0x7f60e41542c0 legacy=0x7f60e4072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60d0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60d00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d0004d10 0x7f60d0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4159f20 con 0x7f60d0003b40 2024-03-16T20:01:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d000ba40 con 0x7f60d0004d10 2024-03-16T20:01:27.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.844+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d000bbb0 con 0x7f60e4070360 2024-03-16T20:01:27.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.848+0000 7f60e93d1640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60d0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e9bd2640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d0004d10 0x7f60d0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e8bd0640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60d00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e93d1640 1 -- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 msgr2=0x7f60d00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:27.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e93d1640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4070360 0x7f60d00047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:27.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e93d1640 1 -- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d0004d10 msgr2=0x7f60d0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:27.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e93d1640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d0004d10 0x7f60d0005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:27.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.852+0000 7f60e93d1640 1 -- 172.21.15.32:0/796826415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e415a1e0 con 0x7f60d0003b40 2024-03-16T20:01:27.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.856+0000 7f60e93d1640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60d0003f40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f60e000c210 tx=0x7f60e001ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b39410b8af9e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0005ae0 con 0x7f60d0003b40 2024-03-16T20:01:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e0005c40 con 0x7f60d0003b40 2024-03-16T20:01:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60dbfff640 1 -- 172.21.15.32:0/796826415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00a5750 con 0x7f60d0003b40 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- 172.21.15.32:0/796826415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 msgr2=0x7f60d0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 --2- 172.21.15.32:0/796826415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60d0003f40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f60e000c210 tx=0x7f60e001ca60 comp rx=0 tx=0).stop 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60e9bd2640 1 -- 172.21.15.32:0/796826415 reap_dead start 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- 172.21.15.32:0/796826415 shutdown_connections 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- 172.21.15.32:0/796826415 >> 172.21.15.32:0/796826415 conn(0x7f60e406c0d0 msgr2=0x7f60e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- 172.21.15.32:0/796826415 shutdown_connections 2024-03-16T20:01:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- 172.21.15.32:0/796826415 wait complete. 2024-03-16T20:01:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 Processor -- start 2024-03-16T20:01:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.860+0000 7f60eb635640 1 -- start start 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40c4610 0x7f60e414fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4145ea0 0x7f60e4146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40c5aa0 con 0x7f60d0003b40 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4146860 con 0x7f60e40c4610 2024-03-16T20:01:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60eb635640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e414ee60 con 0x7f60e4145ea0 2024-03-16T20:01:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60e93d1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60e93d1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32968/0 (socket says 172.21.15.32:32968) 2024-03-16T20:01:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60e93d1640 1 -- 172.21.15.32:0/1038383584 learned_addr learned my addr 172.21.15.32:0/1038383584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60e8bd0640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40c4610 0x7f60e414fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.864+0000 7f60e9bd2640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4145ea0 0x7f60e4146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 -- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4145ea0 msgr2=0x7f60e4146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4145ea0 0x7f60e4146320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 -- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40c4610 msgr2=0x7f60e414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40c4610 0x7f60e414fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 -- 172.21.15.32:0/1038383584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e414f140 con 0x7f60d0003b40 2024-03-16T20:01:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.868+0000 7f60e93d1640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60e001c890 tx=0x7f60d0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:27.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b4070 con 0x7f60d0003b40 2024-03-16T20:01:27.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e0007090 con 0x7f60d0003b40 2024-03-16T20:01:27.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0007400 con 0x7f60d0003b40 2024-03-16T20:01:27.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60eb635640 1 -- 172.21.15.32:0/1038383584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e40c5cf0 con 0x7f60d0003b40 2024-03-16T20:01:27.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60eb635640 1 -- 172.21.15.32:0/1038383584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e414f420 con 0x7f60d0003b40 2024-03-16T20:01:27.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.876+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c00052a0 con 0x7f60d0003b40 2024-03-16T20:01:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.916+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60e0007560 con 0x7f60d0003b40 2024-03-16T20:01:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.916+0000 7f60d9ffb640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 0x7f60c8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.916+0000 7f60e8bd0640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 0x7f60c8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:27.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:27.916+0000 7f60e8bd0640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 0x7f60c8069350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f60e41509b0 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:28.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.208+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f60e00ef260 con 0x7f60d0003b40 2024-03-16T20:01:28.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.212+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f60e00eb500 con 0x7f60d0003b40 2024-03-16T20:01:28.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.416+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60c0005900 con 0x7f60d0003b40 2024-03-16T20:01:28.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.440+0000 7f60d9ffb640 1 -- 172.21.15.32:0/1038383584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f60c0005900 con 0x7f60d0003b40 2024-03-16T20:01:28.414 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:28.414 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:28.414 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.440+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 msgr2=0x7f60c8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.440+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 0x7f60c8069350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f60e41509b0 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2024-03-16T20:01:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.440+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 msgr2=0x7f60e414f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.440+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60e001c890 tx=0x7f60d0005800 comp rx=0 tx=0).stop 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 shutdown_connections 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066e90 0x7f60c8069350 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e4145ea0 0x7f60e4146320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40c4610 0x7f60e414fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 --2- 172.21.15.32:0/1038383584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d0003b40 0x7f60e414f9a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 >> 172.21.15.32:0/1038383584 conn(0x7f60e406c0d0 msgr2=0x7f60e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 shutdown_connections 2024-03-16T20:01:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:28.444+0000 7f60c77fe640 1 -- 172.21.15.32:0/1038383584 wait complete. 2024-03-16T20:01:29.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:29.042 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:30.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:30.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 Processor -- start 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 -- start start 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07b80c6040 0x7f07b80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f07b80c6420 0x7f07b8157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b8070760 con 0x7f07b806ba30 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80731c0 con 0x7f07b80c6040 2024-03-16T20:01:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.176+0000 7f07bd21f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80734a0 con 0x7f07b80c6420 2024-03-16T20:01:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.180+0000 7f07b6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07b80c6040 0x7f07b80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.180+0000 7f07b6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07b80c6040 0x7f07b80c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42618/0 (socket says 172.21.15.32:42618) 2024-03-16T20:01:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.180+0000 7f07b6d9d640 1 -- 172.21.15.32:0/1925944211 learned_addr learned my addr 172.21.15.32:0/1925944211 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:30.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.180+0000 7f07b659c640 1 --2- 172.21.15.32:0/1925944211 >> v2:172.21.15.59:3300/0 conn(0x7f07b80c6420 0x7f07b8157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.184+0000 7f07b6d9d640 1 -- 172.21.15.32:0/1925944211 >> v2:172.21.15.59:3300/0 conn(0x7f07b80c6420 msgr2=0x7f07b8157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.184+0000 7f07b6d9d640 1 --2- 172.21.15.32:0/1925944211 >> v2:172.21.15.59:3300/0 conn(0x7f07b80c6420 0x7f07b8157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.184+0000 7f07b6d9d640 1 -- 172.21.15.32:0/1925944211 >> v1:172.21.15.32:6789/0 conn(0x7f07b806ba30 legacy=0x7f07b81516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.184+0000 7f07b6d9d640 1 -- 172.21.15.32:0/1925944211 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b806fe30 con 0x7f07b80c6040 2024-03-16T20:01:30.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.192+0000 7f07b6d9d640 1 --2- 172.21.15.32:0/1925944211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80c6040 0x7f07b80c8800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f07b806cf50 tx=0x7f07a801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4facf5ec89d9393 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:30.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.196+0000 7f07b5d9b640 1 -- 172.21.15.32:0/1925944211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80ab600 con 0x7f07b80c6040 2024-03-16T20:01:30.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.196+0000 7f07b5d9b640 1 -- 172.21.15.32:0/1925944211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a80a7050 con 0x7f07b80c6040 2024-03-16T20:01:30.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.196+0000 7f07b5d9b640 1 -- 172.21.15.32:0/1925944211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80ab980 con 0x7f07b80c6040 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- 172.21.15.32:0/1925944211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80c6040 msgr2=0x7f07b80c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- 172.21.15.32:0/1925944211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80c6040 0x7f07b80c8800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f07b806cf50 tx=0x7f07a801c480 comp rx=0 tx=0).stop 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- 172.21.15.32:0/1925944211 shutdown_connections 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- 172.21.15.32:0/1925944211 >> v2:172.21.15.59:3300/0 conn(0x7f07b80c6420 0x7f07b8157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- 172.21.15.32:0/1925944211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80c6040 0x7f07b80c8800 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- 172.21.15.32:0/1925944211 >> 172.21.15.32:0/1925944211 conn(0x7f07b806c0d0 msgr2=0x7f07b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- 172.21.15.32:0/1925944211 shutdown_connections 2024-03-16T20:01:30.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- 172.21.15.32:0/1925944211 wait complete. 2024-03-16T20:01:30.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 Processor -- start 2024-03-16T20:01:30.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- start start 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f07b80c6040 0x7f07b813f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b80c6420 0x7f07b8156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8155dc0 con 0x7f07b806ba30 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81f7070 con 0x7f07b80c6420 2024-03-16T20:01:30.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07bd21f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80c49b0 con 0x7f07b80c6040 2024-03-16T20:01:30.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07b6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07b6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42624/0 (socket says 172.21.15.32:42624) 2024-03-16T20:01:30.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.200+0000 7f07b6d9d640 1 -- 172.21.15.32:0/527612276 learned_addr learned my addr 172.21.15.32:0/527612276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:30.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b759e640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b80c6420 0x7f07b8156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b659c640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f07b80c6040 0x7f07b813f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b6d9d640 1 -- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f07b80c6040 msgr2=0x7f07b813f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b6d9d640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f07b80c6040 0x7f07b813f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b6d9d640 1 -- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b80c6420 msgr2=0x7f07b8156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b6d9d640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b80c6420 0x7f07b8156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.204+0000 7f07b6d9d640 1 -- 172.21.15.32:0/527612276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80094d0 con 0x7f07b806ba30 2024-03-16T20:01:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.212+0000 7f07b6d9d640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f07a80ae040 tx=0x7f07a801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80b6420 con 0x7f07b806ba30 2024-03-16T20:01:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a80b6580 con 0x7f07b806ba30 2024-03-16T20:01:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f07bd21f640 1 -- 172.21.15.32:0/527612276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b80c4c50 con 0x7f07b806ba30 2024-03-16T20:01:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80b6910 con 0x7f07b806ba30 2024-03-16T20:01:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f07bd21f640 1 -- 172.21.15.32:0/527612276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b80c5210 con 0x7f07b806ba30 2024-03-16T20:01:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.216+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07900052a0 con 0x7f07b806ba30 2024-03-16T20:01:30.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.504+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f07a80b2040 con 0x7f07b806ba30 2024-03-16T20:01:30.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.508+0000 7f079f7fe640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 0x7f07a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:30.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.508+0000 7f07b659c640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 0x7f07a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:30.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.508+0000 7f07b659c640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 0x7f07a4069330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f07b8149ad0 tx=0x7f07ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.532+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f07a80efda0 con 0x7f07b806ba30 2024-03-16T20:01:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.532+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f07a80ec040 con 0x7f07b806ba30 2024-03-16T20:01:30.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.780+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0790005900 con 0x7f07b806ba30 2024-03-16T20:01:30.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.872+0000 7f079f7fe640 1 -- 172.21.15.32:0/527612276 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f0790005900 con 0x7f07b806ba30 2024-03-16T20:01:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:30.847 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 msgr2=0x7f07a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 0x7f07a4069330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f07b8149ad0 tx=0x7f07ac008040 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 msgr2=0x7f07b8157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f07a80ae040 tx=0x7f07a801acc0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 shutdown_connections 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f07a4066e70 0x7f07a4069330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b80c6420 0x7f07b8156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f07b80c6040 0x7f07b813f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 --2- 172.21.15.32:0/527612276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b806ba30 0x7f07b8157a70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 >> 172.21.15.32:0/527612276 conn(0x7f07b806c0d0 msgr2=0x7f07b806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 shutdown_connections 2024-03-16T20:01:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:30.876+0000 7f079d7fa640 1 -- 172.21.15.32:0/527612276 wait complete. 2024-03-16T20:01:31.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:31.472 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:32.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:32.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 Processor -- start 2024-03-16T20:01:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 -- start start 2024-03-16T20:01:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84a4157180 0x7f84a406be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f84a40b1ea0 0x7f84a4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a406bca0 con 0x7f84a4150710 2024-03-16T20:01:32.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a4156cf0 con 0x7f84a4157180 2024-03-16T20:01:32.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84abbb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a406c500 con 0x7f84a40b1ea0 2024-03-16T20:01:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.604+0000 7f84a9150640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84a4157180 0x7f84a406be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84a4157180 0x7f84a406be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42626/0 (socket says 172.21.15.32:42626) 2024-03-16T20:01:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 -- 172.21.15.32:0/1902941226 learned_addr learned my addr 172.21.15.32:0/1902941226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84aa152640 1 --2- 172.21.15.32:0/1902941226 >> v2:172.21.15.59:3300/0 conn(0x7f84a40b1ea0 0x7f84a4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 -- 172.21.15.32:0/1902941226 >> v2:172.21.15.59:3300/0 conn(0x7f84a40b1ea0 msgr2=0x7f84a4072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 --2- 172.21.15.32:0/1902941226 >> v2:172.21.15.59:3300/0 conn(0x7f84a40b1ea0 0x7f84a4072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 -- 172.21.15.32:0/1902941226 >> v1:172.21.15.59:6789/0 conn(0x7f84a4150710 legacy=0x7f84a406d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.608+0000 7f84a9150640 1 -- 172.21.15.32:0/1902941226 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a406c7a0 con 0x7f84a4157180 2024-03-16T20:01:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.612+0000 7f84a9150640 1 --2- 172.21.15.32:0/1902941226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a4157180 0x7f84a406be70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8494009870 tx=0x7f849401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397c3bf1b096e838 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:32.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.616+0000 7f84a894f640 1 -- 172.21.15.32:0/1902941226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84940ac630 con 0x7f84a4157180 2024-03-16T20:01:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84a894f640 1 -- 172.21.15.32:0/1902941226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84940ac790 con 0x7f84a4157180 2024-03-16T20:01:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84a894f640 1 -- 172.21.15.32:0/1902941226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84940acb20 con 0x7f84a4157180 2024-03-16T20:01:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- 172.21.15.32:0/1902941226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a4157180 msgr2=0x7f84a406be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 --2- 172.21.15.32:0/1902941226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a4157180 0x7f84a406be70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8494009870 tx=0x7f849401c4e0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- 172.21.15.32:0/1902941226 shutdown_connections 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 --2- 172.21.15.32:0/1902941226 >> v2:172.21.15.59:3300/0 conn(0x7f84a40b1ea0 0x7f84a4072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 --2- 172.21.15.32:0/1902941226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a4157180 0x7f84a406be70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- 172.21.15.32:0/1902941226 >> 172.21.15.32:0/1902941226 conn(0x7f84a406afc0 msgr2=0x7f84a406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- 172.21.15.32:0/1902941226 shutdown_connections 2024-03-16T20:01:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- 172.21.15.32:0/1902941226 wait complete. 2024-03-16T20:01:32.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 Processor -- start 2024-03-16T20:01:32.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.620+0000 7f84abbb5640 1 -- start start 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84a4150710 0x7f84a41f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a4157180 0x7f84a414abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a41422c0 con 0x7f84a40b1ea0 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a41f36e0 con 0x7f84a4157180 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84abbb5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40c3480 con 0x7f84a4150710 2024-03-16T20:01:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84a9951640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84a9951640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42638/0 (socket says 172.21.15.32:42638) 2024-03-16T20:01:32.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84a9951640 1 -- 172.21.15.32:0/4081731385 learned_addr learned my addr 172.21.15.32:0/4081731385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:32.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84aa152640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a4157180 0x7f84a414abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.624+0000 7f84a9150640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84a4150710 0x7f84a41f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.628+0000 7f84a9951640 1 -- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84a4150710 msgr2=0x7f84a41f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:32.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.628+0000 7f84a9951640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84a4150710 0x7f84a41f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.628+0000 7f84a9951640 1 -- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a4157180 msgr2=0x7f84a414abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:32.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.628+0000 7f84a9951640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a4157180 0x7f84a414abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:32.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.628+0000 7f84a9951640 1 -- 172.21.15.32:0/4081731385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940094d0 con 0x7f84a40b1ea0 2024-03-16T20:01:32.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.632+0000 7f84a9951640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c490 tx=0x7f84a0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.636+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0003600 con 0x7f84a40b1ea0 2024-03-16T20:01:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.636+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a0003780 con 0x7f84a40b1ea0 2024-03-16T20:01:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.636+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a40c3760 con 0x7f84a40b1ea0 2024-03-16T20:01:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.636+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0003b70 con 0x7f84a40b1ea0 2024-03-16T20:01:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.636+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a41f3c60 con 0x7f84a40b1ea0 2024-03-16T20:01:32.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.640+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4068000 con 0x7f84a40b1ea0 2024-03-16T20:01:32.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.668+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f84a0003cd0 con 0x7f84a40b1ea0 2024-03-16T20:01:32.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.668+0000 7f849a7fc640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 0x7f8488069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.668+0000 7f84a9150640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 0x7f8488069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.668+0000 7f84a9150640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 0x7f8488069350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84940ae000 tx=0x7f84940a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.676+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f84a0055080 con 0x7f84a40b1ea0 2024-03-16T20:01:32.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:32.940+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f84a0051320 con 0x7f84a40b1ea0 2024-03-16T20:01:33.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.344+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a40b1a70 con 0x7f84a40b1ea0 2024-03-16T20:01:33.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.368+0000 7f849a7fc640 1 -- 172.21.15.32:0/4081731385 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f84a0017070 con 0x7f84a40b1ea0 2024-03-16T20:01:33.344 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:33.344 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:33.344 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 msgr2=0x7f8488069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 0x7f8488069350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84940ae000 tx=0x7f84940a7000 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 msgr2=0x7f84a41f4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c490 tx=0x7f84a0001e80 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 shutdown_connections 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8488066e90 0x7f8488069350 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a4157180 0x7f84a414abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84a4150710 0x7f84a41f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 --2- 172.21.15.32:0/4081731385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84a40b1ea0 0x7f84a41f4180 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 >> 172.21.15.32:0/4081731385 conn(0x7f84a406afc0 msgr2=0x7f84a414dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:33.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 shutdown_connections 2024-03-16T20:01:33.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:33.372+0000 7f84abbb5640 1 -- 172.21.15.32:0/4081731385 wait complete. 2024-03-16T20:01:33.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:33.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:34.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.104+0000 7f9cfa863640 1 Processor -- start 2024-03-16T20:01:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.104+0000 7f9cfa863640 1 -- start start 2024-03-16T20:01:35.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cfa863640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cf40c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cfa863640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf4158c80 con 0x7f9cf415b2a0 2024-03-16T20:01:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cfa863640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf415bfc0 con 0x7f9cf4151ef0 2024-03-16T20:01:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cfa863640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40b0ad0 con 0x7f9cf40c81e0 2024-03-16T20:01:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cf8e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9cf415b2a0 0x7f9cf415b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55596/0 (socket says 172.21.15.32:55596) 2024-03-16T20:01:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cf8e00640 1 -- 172.21.15.32:0/1767653746 learned_addr learned my addr 172.21.15.32:0/1767653746 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.108+0000 7f9cf37fe640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cf40c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.112+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009931167 0 0) 0x7f9cf4158c80 con 0x7f9cf415b2a0 2024-03-16T20:01:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.112+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc003530 con 0x7f9cf415b2a0 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475655551 0 0) 0x7f9cdc003530 con 0x7f9cf415b2a0 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 msgr2=0x7f9cf40c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cf40c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 >> v1:172.21.15.59:6789/0 conn(0x7f9cf4151ef0 legacy=0x7f9cf40c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4157ec0 con 0x7f9cf415b2a0 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f9cec001a00 con 0x7f9cf415b2a0 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 >> v1:172.21.15.32:6789/0 conn(0x7f9cf415b2a0 legacy=0x7f9cf415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cdc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40b0ad0 con 0x7f9cdc0039b0 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00bb30 con 0x7f9cdc004d10 2024-03-16T20:01:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00bcf0 con 0x7f9cf40c81e0 2024-03-16T20:01:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.116+0000 7f9cf8e00640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.120+0000 7f9cf3fff640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf37fe640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cdc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf8e00640 1 -- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 msgr2=0x7f9cdc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf8e00640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf40c81e0 0x7f9cdc0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf8e00640 1 -- 172.21.15.32:0/1767653746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc0039b0 msgr2=0x7f9cdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf8e00640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc0039b0 0x7f9cdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.124+0000 7f9cf8e00640 1 -- 172.21.15.32:0/1767653746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc003530 con 0x7f9cdc004d10 2024-03-16T20:01:35.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.128+0000 7f9cf8e00640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9cec002240 tx=0x7f9cec01f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=530f1250b8beaac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.132+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec0ac670 con 0x7f9cdc004d10 2024-03-16T20:01:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cf2ffd640 1 -- 172.21.15.32:0/1767653746 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec0ac7d0 con 0x7f9cdc004d10 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 -- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cdc004d10 msgr2=0x7f9cdc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1767653746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cdc004d10 0x7f9cdc005190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9cec002240 tx=0x7f9cec01f9c0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cf8e00640 1 -- 172.21.15.32:0/1767653746 reap_dead start 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 -- 172.21.15.32:0/1767653746 shutdown_connections 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 -- 172.21.15.32:0/1767653746 >> 172.21.15.32:0/1767653746 conn(0x7f9cf4070f90 msgr2=0x7f9cf40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 -- 172.21.15.32:0/1767653746 shutdown_connections 2024-03-16T20:01:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.136+0000 7f9cfa863640 1 -- 172.21.15.32:0/1767653746 wait complete. 2024-03-16T20:01:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 Processor -- start 2024-03-16T20:01:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 -- start start 2024-03-16T20:01:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cf4150210 0x7f9cf41f0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf41e3a20 0x7f9cf414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf415bde0 con 0x7f9cdc004d10 2024-03-16T20:01:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf41f3f90 con 0x7f9cf4150210 2024-03-16T20:01:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cfa863640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf41f04a0 con 0x7f9cf41e3a20 2024-03-16T20:01:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cf3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cf37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cf4150210 0x7f9cf41f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cf3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42664/0 (socket says 172.21.15.32:42664) 2024-03-16T20:01:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cf3fff640 1 -- 172.21.15.32:0/1349606976 learned_addr learned my addr 172.21.15.32:0/1349606976 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.140+0000 7f9cf8e00640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf41e3a20 0x7f9cf414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.144+0000 7f9cf3fff640 1 -- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf41e3a20 msgr2=0x7f9cf414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.144+0000 7f9cf3fff640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf41e3a20 0x7f9cf414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.144+0000 7f9cf3fff640 1 -- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cf4150210 msgr2=0x7f9cf41f0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.144+0000 7f9cf3fff640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cf4150210 0x7f9cf41f0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.144+0000 7f9cf3fff640 1 -- 172.21.15.32:0/1349606976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf41f0170 con 0x7f9cdc004d10 2024-03-16T20:01:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.148+0000 7f9cf3fff640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400f820 tx=0x7f9ce400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.152+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce40050d0 con 0x7f9cdc004d10 2024-03-16T20:01:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.152+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce4005230 con 0x7f9cdc004d10 2024-03-16T20:01:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.152+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400d730 con 0x7f9cdc004d10 2024-03-16T20:01:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.152+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf406df00 con 0x7f9cdc004d10 2024-03-16T20:01:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.152+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf406dc10 con 0x7f9cdc004d10 2024-03-16T20:01:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.156+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf4068000 con 0x7f9cdc004d10 2024-03-16T20:01:35.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.444+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9ce400d890 con 0x7f9cdc004d10 2024-03-16T20:01:35.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.444+0000 7f9cf0ff9640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 0x7f9ce0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:35.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.444+0000 7f9cf37fe640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 0x7f9ce0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:35.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.444+0000 7f9cf37fe640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 0x7f9ce0069300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9cf41506d0 tx=0x7f9ce8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.468+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9ce4055f70 con 0x7f9cdc004d10 2024-03-16T20:01:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.468+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9ce4052210 con 0x7f9cdc004d10 2024-03-16T20:01:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.768+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cf4070a40 con 0x7f9cdc004d10 2024-03-16T20:01:35.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.832+0000 7f9cf0ff9640 1 -- 172.21.15.32:0/1349606976 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9ce40558d0 con 0x7f9cdc004d10 2024-03-16T20:01:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:35.809 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:00:33.379953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:01:35.810 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 msgr2=0x7f9ce0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 0x7f9ce0069300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9cf41506d0 tx=0x7f9ce8008040 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 msgr2=0x7f9cf414f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400f820 tx=0x7f9ce400fce0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 shutdown_connections 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ce0066e40 0x7f9ce0069300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cf41e3a20 0x7f9cf414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cf4150210 0x7f9cf41f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 --2- 172.21.15.32:0/1349606976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cdc004d10 0x7f9cf414f950 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 >> 172.21.15.32:0/1349606976 conn(0x7f9cf4070f90 msgr2=0x7f9cf4150f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 shutdown_connections 2024-03-16T20:01:35.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:35.836+0000 7f9cfa863640 1 -- 172.21.15.32:0/1349606976 wait complete. 2024-03-16T20:01:36.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:00:33.379953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:01:36.440 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:37.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 Processor -- start 2024-03-16T20:01:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 -- start start 2024-03-16T20:01:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a881543f0 0x7f2a8815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a8806f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8815d5a0 con 0x7f2a8815e250 2024-03-16T20:01:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8815d880 con 0x7f2a881543f0 2024-03-16T20:01:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.572+0000 7f2a90282640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8806ba40 con 0x7f2a880c8bb0 2024-03-16T20:01:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e01e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a881543f0 0x7f2a8815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8d81d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a8806f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e81f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2a8815e250 0x7f2a88072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:37698/0 (socket says 172.21.15.32:37698) 2024-03-16T20:01:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e81f640 1 -- 172.21.15.32:0/3296468147 learned_addr learned my addr 172.21.15.32:0/3296468147 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e01e640 1 -- 172.21.15.32:0/3296468147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 msgr2=0x7f2a8806f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e01e640 1 --2- 172.21.15.32:0/3296468147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a8806f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e01e640 1 -- 172.21.15.32:0/3296468147 >> v1:172.21.15.59:6789/0 conn(0x7f2a8815e250 legacy=0x7f2a88072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.576+0000 7f2a8e01e640 1 -- 172.21.15.32:0/3296468147 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8815ae70 con 0x7f2a881543f0 2024-03-16T20:01:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.580+0000 7f2a8e01e640 1 --2- 172.21.15.32:0/3296468147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a8815dd10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2a840097c0 tx=0x7f2a8401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e719a33eec09681 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a8d01c640 1 -- 172.21.15.32:0/3296468147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840a8990 con 0x7f2a881543f0 2024-03-16T20:01:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a8d01c640 1 -- 172.21.15.32:0/3296468147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a840a8af0 con 0x7f2a881543f0 2024-03-16T20:01:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a8d01c640 1 -- 172.21.15.32:0/3296468147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840a8e80 con 0x7f2a881543f0 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- 172.21.15.32:0/3296468147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 msgr2=0x7f2a8815dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- 172.21.15.32:0/3296468147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a8815dd10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2a840097c0 tx=0x7f2a8401ec90 comp rx=0 tx=0).stop 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- 172.21.15.32:0/3296468147 shutdown_connections 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- 172.21.15.32:0/3296468147 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a8806f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- 172.21.15.32:0/3296468147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a8815dd10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- 172.21.15.32:0/3296468147 >> 172.21.15.32:0/3296468147 conn(0x7f2a880cc7a0 msgr2=0x7f2a880ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- 172.21.15.32:0/3296468147 shutdown_connections 2024-03-16T20:01:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- 172.21.15.32:0/3296468147 wait complete. 2024-03-16T20:01:37.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 Processor -- start 2024-03-16T20:01:37.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- start start 2024-03-16T20:01:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a881f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a8815e250 0x7f2a88151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a881f06d0 con 0x7f2a881543f0 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a881f00c0 con 0x7f2a8815e250 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.588+0000 7f2a90282640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880ca3d0 con 0x7f2a880c8bb0 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.592+0000 7f2a8d81d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.592+0000 7f2a8d81d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42698/0 (socket says 172.21.15.32:42698) 2024-03-16T20:01:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.592+0000 7f2a8d81d640 1 -- 172.21.15.32:0/541885912 learned_addr learned my addr 172.21.15.32:0/541885912 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:37.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.592+0000 7f2a8e81f640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a8815e250 0x7f2a88151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.592+0000 7f2a8e01e640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a881f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.596+0000 7f2a8d81d640 1 -- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 msgr2=0x7f2a881f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:37.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.596+0000 7f2a8d81d640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a881f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:37.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.596+0000 7f2a8d81d640 1 -- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a8815e250 msgr2=0x7f2a88151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:37.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.596+0000 7f2a8d81d640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a8815e250 0x7f2a88151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:37.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.596+0000 7f2a8d81d640 1 -- 172.21.15.32:0/541885912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84009420 con 0x7f2a881543f0 2024-03-16T20:01:37.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.600+0000 7f2a8d81d640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800e900 tx=0x7f2a7800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:37.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.604+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78009690 con 0x7f2a881543f0 2024-03-16T20:01:37.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.604+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7800ca00 con 0x7f2a881543f0 2024-03-16T20:01:37.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.604+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a880ca6b0 con 0x7f2a881543f0 2024-03-16T20:01:37.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.604+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7800cdb0 con 0x7f2a881543f0 2024-03-16T20:01:37.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.604+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a881522b0 con 0x7f2a881543f0 2024-03-16T20:01:37.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.608+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a881eff20 con 0x7f2a881543f0 2024-03-16T20:01:37.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.632+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2a78013470 con 0x7f2a881543f0 2024-03-16T20:01:37.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.632+0000 7f2a7effd640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 0x7f2a6c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:37.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.632+0000 7f2a8e01e640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 0x7f2a6c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:37.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.632+0000 7f2a8e01e640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 0x7f2a6c069300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a840b2000 tx=0x7f2a84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.928+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2a78055e10 con 0x7f2a881543f0 2024-03-16T20:01:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:37.928+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2a780520b0 con 0x7f2a881543f0 2024-03-16T20:01:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.132+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a881eff20 con 0x7f2a881543f0 2024-03-16T20:01:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.156+0000 7f2a7effd640 1 -- 172.21.15.32:0/541885912 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f2a881eff20 con 0x7f2a881543f0 2024-03-16T20:01:38.133 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:38.133 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 msgr2=0x7f2a6c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 0x7f2a6c069300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a840b2000 tx=0x7f2a84008040 comp rx=0 tx=0).stop 2024-03-16T20:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 msgr2=0x7f2a881e5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800e900 tx=0x7f2a7800edc0 comp rx=0 tx=0).stop 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 shutdown_connections 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a6c066e40 0x7f2a6c069300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a8815e250 0x7f2a88151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a881543f0 0x7f2a881e5700 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 --2- 172.21.15.32:0/541885912 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a880c8bb0 0x7f2a881f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 >> 172.21.15.32:0/541885912 conn(0x7f2a880cc7a0 msgr2=0x7f2a88153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 shutdown_connections 2024-03-16T20:01:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:38.160+0000 7f2a90282640 1 -- 172.21.15.32:0/541885912 wait complete. 2024-03-16T20:01:38.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:38.762 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:39.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.896+0000 7fa021790640 1 Processor -- start 2024-03-16T20:01:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.896+0000 7fa021790640 1 -- start start 2024-03-16T20:01:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.896+0000 7fa021790640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa01c0c5600 0x7fa01c066ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa021790640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa01c0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa021790640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c0cccb0 con 0x7fa01c0c69e0 2024-03-16T20:01:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa021790640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0c7000 con 0x7fa01c0c5600 2024-03-16T20:01:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa021790640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c157df0 con 0x7fa01c15b9e0 2024-03-16T20:01:39.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa01a7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa01c0c69e0 0x7fa01c15b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49048/0 (socket says 172.21.15.32:49048) 2024-03-16T20:01:39.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa01a7fc640 1 -- 172.21.15.32:0/3621923960 learned_addr learned my addr 172.21.15.32:0/3621923960 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:39.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa01c0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.900+0000 7fa01affd640 1 --2- 172.21.15.32:0/3621923960 >> v2:172.21.15.59:3300/0 conn(0x7fa01c0c5600 0x7fa01c066ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.904+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925765200 0 0) 0x7fa01c0cccb0 con 0x7fa01c0c69e0 2024-03-16T20:01:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.904+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa004003530 con 0x7fa01c0c69e0 2024-03-16T20:01:39.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661315470 0 0) 0x7fa004003530 con 0x7fa01c0c69e0 2024-03-16T20:01:39.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 msgr2=0x7fa01c0c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa01c0c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 >> v2:172.21.15.59:3300/0 conn(0x7fa01c0c5600 msgr2=0x7fa01c066ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 --2- 172.21.15.32:0/3621923960 >> v2:172.21.15.59:3300/0 conn(0x7fa01c0c5600 0x7fa01c066ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c158090 con 0x7fa01c0c69e0 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fa010001680 con 0x7fa01c0c69e0 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 >> v1:172.21.15.32:6789/0 conn(0x7fa01c0c69e0 legacy=0x7fa01c15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa004003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004003f50 0x7fa0040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa004004d10 0x7fa004005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0c7000 con 0x7fa004004d10 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c157df0 con 0x7fa004003f50 2024-03-16T20:01:39.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00400ba10 con 0x7fa01c15b9e0 2024-03-16T20:01:39.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.908+0000 7fa01a7fc640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004003f50 0x7fa0040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa004004d10 0x7fa004005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01affd640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa004003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 -- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 msgr2=0x7fa004003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c15b9e0 0x7fa004003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 -- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004003f50 msgr2=0x7fa0040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004003f50 0x7fa0040047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.912+0000 7fa01b7fe640 1 -- 172.21.15.32:0/3621923960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004003530 con 0x7fa004004d10 2024-03-16T20:01:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.920+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa004004d10 0x7fa004005190 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa014009fc0 tx=0x7fa01401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36896fca97efab46 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.924+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140a5c90 con 0x7fa004004d10 2024-03-16T20:01:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.924+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01400f040 con 0x7fa004004d10 2024-03-16T20:01:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.924+0000 7fa019ffb640 1 -- 172.21.15.32:0/3621923960 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140ad680 con 0x7fa004004d10 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- 172.21.15.32:0/3621923960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa004004d10 msgr2=0x7fa004005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 --2- 172.21.15.32:0/3621923960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa004004d10 0x7fa004005190 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa014009fc0 tx=0x7fa01401cc80 comp rx=0 tx=0).stop 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa01b7fe640 1 -- 172.21.15.32:0/3621923960 reap_dead start 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- 172.21.15.32:0/3621923960 shutdown_connections 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- 172.21.15.32:0/3621923960 >> 172.21.15.32:0/3621923960 conn(0x7fa01c0cbfe0 msgr2=0x7fa01c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- 172.21.15.32:0/3621923960 shutdown_connections 2024-03-16T20:01:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- 172.21.15.32:0/3621923960 wait complete. 2024-03-16T20:01:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 Processor -- start 2024-03-16T20:01:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- start start 2024-03-16T20:01:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004004d10 0x7fa01c14b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c152ed0 0x7fa01c1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c156c40 con 0x7fa01c157800 2024-03-16T20:01:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c155b50 con 0x7fa004004d10 2024-03-16T20:01:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.928+0000 7fa021790640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1eb300 con 0x7fa01c152ed0 2024-03-16T20:01:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.932+0000 7fa01a7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.932+0000 7fa01a7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34354/0 (socket says 172.21.15.32:34354) 2024-03-16T20:01:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.932+0000 7fa01a7fc640 1 -- 172.21.15.32:0/2540980246 learned_addr learned my addr 172.21.15.32:0/2540980246 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.932+0000 7fa01affd640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004004d10 0x7fa01c14b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.932+0000 7fa01b7fe640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c152ed0 0x7fa01c1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.936+0000 7fa01a7fc640 1 -- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c152ed0 msgr2=0x7fa01c1545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.936+0000 7fa01a7fc640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c152ed0 0x7fa01c1545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.936+0000 7fa01a7fc640 1 -- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004004d10 msgr2=0x7fa01c14b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.936+0000 7fa01a7fc640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004004d10 0x7fa01c14b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.936+0000 7fa01a7fc640 1 -- 172.21.15.32:0/2540980246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c155dc0 con 0x7fa01c157800 2024-03-16T20:01:39.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.940+0000 7fa01a7fc640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa010001ec0 tx=0x7fa01000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.944+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0100074e0 con 0x7fa01c157800 2024-03-16T20:01:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.944+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010007640 con 0x7fa01c157800 2024-03-16T20:01:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.944+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c1560a0 con 0x7fa01c157800 2024-03-16T20:01:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.944+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010004a10 con 0x7fa01c157800 2024-03-16T20:01:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.944+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1eb8a0 con 0x7fa01c157800 2024-03-16T20:01:39.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.948+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c068000 con 0x7fa01c157800 2024-03-16T20:01:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.976+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa01000b6b0 con 0x7fa01c157800 2024-03-16T20:01:39.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.976+0000 7fa0037fe640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 0x7fa008069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:39.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.976+0000 7fa01affd640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 0x7fa008069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:39.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:39.976+0000 7fa01affd640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 0x7fa008069380 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009a40 tx=0x7fa00c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:40.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.300+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa010055690 con 0x7fa01c157800 2024-03-16T20:01:40.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.300+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa010051930 con 0x7fa01c157800 2024-03-16T20:01:40.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.500+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa01c15b7c0 con 0x7fa01c157800 2024-03-16T20:01:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa0037fe640 1 -- 172.21.15.32:0/2540980246 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa010019020 con 0x7fa01c157800 2024-03-16T20:01:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:40.499 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 msgr2=0x7fa008069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 0x7fa008069380 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009a40 tx=0x7fa00c008040 comp rx=0 tx=0).stop 2024-03-16T20:01:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 msgr2=0x7fa01c155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa010001ec0 tx=0x7fa01000dc90 comp rx=0 tx=0).stop 2024-03-16T20:01:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 shutdown_connections 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa008066ec0 0x7fa008069380 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa01c152ed0 0x7fa01c1545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c157800 0x7fa01c155610 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 --2- 172.21.15.32:0/2540980246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa004004d10 0x7fa01c14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 >> 172.21.15.32:0/2540980246 conn(0x7fa01c0cbfe0 msgr2=0x7fa01c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 shutdown_connections 2024-03-16T20:01:40.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:40.524+0000 7fa021790640 1 -- 172.21.15.32:0/2540980246 wait complete. 2024-03-16T20:01:41.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:41.128 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:42.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:01:42.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:01:42.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:42.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.260+0000 7fe8edf49640 1 Processor -- start 2024-03-16T20:01:42.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.260+0000 7fe8edf49640 1 -- start start 2024-03-16T20:01:42.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8edf49640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe8e80c6420 0x7fe8e80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8edf49640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8edf49640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e8070760 con 0x7fe8e80c6040 2024-03-16T20:01:42.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8edf49640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e80731c0 con 0x7fe8e80c6420 2024-03-16T20:01:42.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8edf49640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80734a0 con 0x7fe8e8158390 2024-03-16T20:01:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8e6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe8e80c6420 0x7fe8e80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:57312/0 (socket says 172.21.15.32:57312) 2024-03-16T20:01:42.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.264+0000 7fe8e7fff640 1 -- 172.21.15.32:0/2862037830 learned_addr learned my addr 172.21.15.32:0/2862037830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.268+0000 7fe8e7fff640 1 -- 172.21.15.32:0/2862037830 >> v2:172.21.15.59:3300/0 conn(0x7fe8e80c6420 msgr2=0x7fe8e80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.268+0000 7fe8e7fff640 1 --2- 172.21.15.32:0/2862037830 >> v2:172.21.15.59:3300/0 conn(0x7fe8e80c6420 0x7fe8e80c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.268+0000 7fe8e7fff640 1 -- 172.21.15.32:0/2862037830 >> v1:172.21.15.59:6789/0 conn(0x7fe8e80c6040 legacy=0x7fe8e8157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.268+0000 7fe8e7fff640 1 -- 172.21.15.32:0/2862037830 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e806fe30 con 0x7fe8e8158390 2024-03-16T20:01:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.272+0000 7fe8e7fff640 1 --2- 172.21.15.32:0/2862037830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009e50 tx=0x7fe8e001ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=488ccdf48618f731 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8e67fc640 1 -- 172.21.15.32:0/2862037830 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00a58d0 con 0x7fe8e8158390 2024-03-16T20:01:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8e67fc640 1 -- 172.21.15.32:0/2862037830 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e00a5a30 con 0x7fe8e8158390 2024-03-16T20:01:42.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8e67fc640 1 -- 172.21.15.32:0/2862037830 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00a5dc0 con 0x7fe8e8158390 2024-03-16T20:01:42.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 -- 172.21.15.32:0/2862037830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 msgr2=0x7fe8e81516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 --2- 172.21.15.32:0/2862037830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009e50 tx=0x7fe8e001ca00 comp rx=0 tx=0).stop 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 -- 172.21.15.32:0/2862037830 shutdown_connections 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 --2- 172.21.15.32:0/2862037830 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81516d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 --2- 172.21.15.32:0/2862037830 >> v2:172.21.15.59:3300/0 conn(0x7fe8e80c6420 0x7fe8e80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 -- 172.21.15.32:0/2862037830 >> 172.21.15.32:0/2862037830 conn(0x7fe8e806c0d0 msgr2=0x7fe8e806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 -- 172.21.15.32:0/2862037830 shutdown_connections 2024-03-16T20:01:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.280+0000 7fe8edf49640 1 -- 172.21.15.32:0/2862037830 wait complete. 2024-03-16T20:01:42.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 Processor -- start 2024-03-16T20:01:42.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 -- start start 2024-03-16T20:01:42.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 0x7fe8e81445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80c6420 0x7fe8e81425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e81f1050 con 0x7fe8e80c6040 2024-03-16T20:01:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e81f52c0 con 0x7fe8e80c6420 2024-03-16T20:01:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8edf49640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e81f4740 con 0x7fe8e8158390 2024-03-16T20:01:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8e77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 0x7fe8e81445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81f14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:57328/0 (socket says 172.21.15.32:57328) 2024-03-16T20:01:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.288+0000 7fe8e7fff640 1 -- 172.21.15.32:0/247971039 learned_addr learned my addr 172.21.15.32:0/247971039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.284+0000 7fe8e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80c6420 0x7fe8e81425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.292+0000 7fe8e77fe640 1 -- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 msgr2=0x7fe8e81f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.292+0000 7fe8e77fe640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.292+0000 7fe8e77fe640 1 -- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80c6420 msgr2=0x7fe8e81425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.292+0000 7fe8e77fe640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80c6420 0x7fe8e81425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.292+0000 7fe8e77fe640 1 -- 172.21.15.32:0/247971039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0009b30 con 0x7fe8e80c6040 2024-03-16T20:01:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.296+0000 7fe8e77fe640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 0x7fe8e81445e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800c3c0 tx=0x7fe8d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d80052d0 con 0x7fe8e80c6040 2024-03-16T20:01:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d8005430 con 0x7fe8e80c6040 2024-03-16T20:01:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e81f4a20 con 0x7fe8e80c6040 2024-03-16T20:01:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8007260 con 0x7fe8e80c6040 2024-03-16T20:01:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e81f4fe0 con 0x7fe8e80c6040 2024-03-16T20:01:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.300+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e8074740 con 0x7fe8e80c6040 2024-03-16T20:01:42.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.320+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe8d80073c0 con 0x7fe8e80c6040 2024-03-16T20:01:42.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.324+0000 7fe8cffff640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 0x7fe8d4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:42.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.324+0000 7fe8e6ffd640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 0x7fe8d4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:42.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.324+0000 7fe8e6ffd640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 0x7fe8d4069300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8e81559a0 tx=0x7fe8dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.608+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe8d8007670 con 0x7fe8e80c6040 2024-03-16T20:01:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.608+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe8d8051410 con 0x7fe8e80c6040 2024-03-16T20:01:42.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.812+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8e806fa50 con 0x7fe8e80c6040 2024-03-16T20:01:42.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.836+0000 7fe8cffff640 1 -- 172.21.15.32:0/247971039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fe8d8020020 con 0x7fe8e80c6040 2024-03-16T20:01:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:42.811 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:42.811 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.836+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 msgr2=0x7fe8d4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.836+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 0x7fe8d4069300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8e81559a0 tx=0x7fe8dc009190 comp rx=0 tx=0).stop 2024-03-16T20:01:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.836+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 msgr2=0x7fe8e81445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.836+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 0x7fe8e81445e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800c3c0 tx=0x7fe8d800c880 comp rx=0 tx=0).stop 2024-03-16T20:01:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 shutdown_connections 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe8d4066e40 0x7fe8d4069300 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8e8158390 0x7fe8e81f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80c6420 0x7fe8e81425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 --2- 172.21.15.32:0/247971039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e80c6040 0x7fe8e81445e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 >> 172.21.15.32:0/247971039 conn(0x7fe8e806c0d0 msgr2=0x7fe8e814a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 shutdown_connections 2024-03-16T20:01:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:42.840+0000 7fe8edf49640 1 -- 172.21.15.32:0/247971039 wait complete. 2024-03-16T20:01:43.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:43.444 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:44.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 Processor -- start 2024-03-16T20:01:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 -- start start 2024-03-16T20:01:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b4151ef0 0x7f55b40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c97b0 0x7f55b40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b4158c80 con 0x7f55b40c8720 2024-03-16T20:01:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40b0ad0 con 0x7f55b4151ef0 2024-03-16T20:01:44.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55bbd94640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4157cb0 con 0x7f55b40c97b0 2024-03-16T20:01:44.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55b9b30640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b4151ef0 0x7f55b40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55b932f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c97b0 0x7f55b40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55b9b30640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b4151ef0 0x7f55b40c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34366/0 (socket says 172.21.15.32:34366) 2024-03-16T20:01:44.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.576+0000 7f55b9b30640 1 -- 172.21.15.32:0/3790049847 learned_addr learned my addr 172.21.15.32:0/3790049847 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:44.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.580+0000 7f55b9b30640 1 -- 172.21.15.32:0/3790049847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c97b0 msgr2=0x7f55b40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:44.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.580+0000 7f55b9b30640 1 --2- 172.21.15.32:0/3790049847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c97b0 0x7f55b40c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:44.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.580+0000 7f55b9b30640 1 -- 172.21.15.32:0/3790049847 >> v1:172.21.15.32:6789/0 conn(0x7f55b40c8720 legacy=0x7f55b415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:44.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.580+0000 7f55b9b30640 1 -- 172.21.15.32:0/3790049847 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b415bfc0 con 0x7f55b4151ef0 2024-03-16T20:01:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.584+0000 7f55b9b30640 1 --2- 172.21.15.32:0/3790049847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b40c9270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009870 tx=0x7f55b001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fceae40cce9c6853 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.588+0000 7f55b8b2e640 1 -- 172.21.15.32:0/3790049847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b00a89d0 con 0x7f55b4151ef0 2024-03-16T20:01:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.588+0000 7f55b8b2e640 1 -- 172.21.15.32:0/3790049847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b00a8b30 con 0x7f55b4151ef0 2024-03-16T20:01:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.588+0000 7f55b8b2e640 1 -- 172.21.15.32:0/3790049847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b00b0440 con 0x7f55b4151ef0 2024-03-16T20:01:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- 172.21.15.32:0/3790049847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 msgr2=0x7f55b40c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3790049847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b40c9270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009870 tx=0x7f55b001ed10 comp rx=0 tx=0).stop 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- 172.21.15.32:0/3790049847 shutdown_connections 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3790049847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c97b0 0x7f55b40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3790049847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b40c9270 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- 172.21.15.32:0/3790049847 >> 172.21.15.32:0/3790049847 conn(0x7f55b4070f90 msgr2=0x7f55b40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- 172.21.15.32:0/3790049847 shutdown_connections 2024-03-16T20:01:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- 172.21.15.32:0/3790049847 wait complete. 2024-03-16T20:01:44.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 Processor -- start 2024-03-16T20:01:44.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- start start 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c8720 0x7f55b414ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b40c97b0 0x7f55b41f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b414cf90 con 0x7f55b4151ef0 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f1b60 con 0x7f55b40c97b0 2024-03-16T20:01:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.592+0000 7f55bbd94640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b414d7c0 con 0x7f55b40c8720 2024-03-16T20:01:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.596+0000 7f55ba331640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.596+0000 7f55b9b30640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c8720 0x7f55b414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.596+0000 7f55b932f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b40c97b0 0x7f55b41f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.596+0000 7f55ba331640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34378/0 (socket says 172.21.15.32:34378) 2024-03-16T20:01:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.596+0000 7f55ba331640 1 -- 172.21.15.32:0/3624922840 learned_addr learned my addr 172.21.15.32:0/3624922840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.600+0000 7f55ba331640 1 -- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c8720 msgr2=0x7f55b414ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.600+0000 7f55ba331640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c8720 0x7f55b414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.600+0000 7f55ba331640 1 -- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b40c97b0 msgr2=0x7f55b41f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.600+0000 7f55ba331640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b40c97b0 0x7f55b41f1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.600+0000 7f55ba331640 1 -- 172.21.15.32:0/3624922840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b00094d0 con 0x7f55b4151ef0 2024-03-16T20:01:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.604+0000 7f55ba331640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c780 tx=0x7f55ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003d90 con 0x7f55b4151ef0 2024-03-16T20:01:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac00ce90 con 0x7f55b4151ef0 2024-03-16T20:01:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b414daa0 con 0x7f55b4151ef0 2024-03-16T20:01:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0052b0 con 0x7f55b4151ef0 2024-03-16T20:01:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b414bf70 con 0x7f55b4151ef0 2024-03-16T20:01:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.608+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4068000 con 0x7f55b4151ef0 2024-03-16T20:01:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.632+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f55ac005410 con 0x7f55b4151ef0 2024-03-16T20:01:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.632+0000 7f55aa7fc640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 0x7f5598069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.636+0000 7f55b9b30640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 0x7f5598069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.636+0000 7f55b9b30640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 0x7f5598069350 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55b00b2000 tx=0x7f55b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:44.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.916+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f55ac055290 con 0x7f55b4151ef0 2024-03-16T20:01:44.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:44.916+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f55ac051530 con 0x7f55b4151ef0 2024-03-16T20:01:45.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.252+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b41553f0 con 0x7f55b4151ef0 2024-03-16T20:01:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55aa7fc640 1 -- 172.21.15.32:0/3624922840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f55ac017020 con 0x7f55b4151ef0 2024-03-16T20:01:45.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:45.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 msgr2=0x7f5598069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 0x7f5598069350 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55b00b2000 tx=0x7f55b0008040 comp rx=0 tx=0).stop 2024-03-16T20:01:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 msgr2=0x7f55b414b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c780 tx=0x7f55ac00cc40 comp rx=0 tx=0).stop 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 shutdown_connections 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5598066e90 0x7f5598069350 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b4151ef0 0x7f55b414b480 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b40c97b0 0x7f55b41f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 --2- 172.21.15.32:0/3624922840 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55b40c8720 0x7f55b414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 >> 172.21.15.32:0/3624922840 conn(0x7f55b4070f90 msgr2=0x7f55b4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 shutdown_connections 2024-03-16T20:01:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:45.272+0000 7f55bbd94640 1 -- 172.21.15.32:0/3624922840 wait complete. 2024-03-16T20:01:45.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:45.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:46.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 Processor -- start 2024-03-16T20:01:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 -- start start 2024-03-16T20:01:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec151ef0 0x7fa5ec0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5ec15b2a0 0x7fa5ec15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec158c80 con 0x7fa5ec0c97b0 2024-03-16T20:01:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec0b0ad0 con 0x7fa5ec15b2a0 2024-03-16T20:01:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.008+0000 7fa5f143b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec157cb0 con 0x7fa5ec151ef0 2024-03-16T20:01:46.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.012+0000 7fa5ea7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa5ec0c97b0 0x7fa5ec0c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49074/0 (socket says 172.21.15.32:49074) 2024-03-16T20:01:46.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.012+0000 7fa5ea7fc640 1 -- 172.21.15.32:0/3180362317 learned_addr learned my addr 172.21.15.32:0/3180362317 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:46.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.012+0000 7fa5eb7fe640 1 --2- 172.21.15.32:0/3180362317 >> v2:172.21.15.59:3300/0 conn(0x7fa5ec15b2a0 0x7fa5ec15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:46.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.012+0000 7fa5eaffd640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec151ef0 0x7fa5ec0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:46.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5e9ffb640 1 -- 172.21.15.32:0/3180362317 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410395327 0 0) 0x7fa5ec158c80 con 0x7fa5ec0c97b0 2024-03-16T20:01:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5e9ffb640 1 -- 172.21.15.32:0/3180362317 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d4003530 con 0x7fa5ec0c97b0 2024-03-16T20:01:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5eb7fe640 1 -- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec151ef0 msgr2=0x7fa5ec0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5eb7fe640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec151ef0 0x7fa5ec0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5eb7fe640 1 -- 172.21.15.32:0/3180362317 >> v1:172.21.15.32:6789/0 conn(0x7fa5ec0c97b0 legacy=0x7fa5ec0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.016+0000 7fa5eb7fe640 1 -- 172.21.15.32:0/3180362317 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec15bfc0 con 0x7fa5ec15b2a0 2024-03-16T20:01:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.020+0000 7fa5eb7fe640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec15b6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009d90 tx=0x7fa5e401f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=312091135da79fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:47.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5e9ffb640 1 -- 172.21.15.32:0/3180362317 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40af600 con 0x7fa5ec15b2a0 2024-03-16T20:01:47.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5e9ffb640 1 -- 172.21.15.32:0/3180362317 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e40ab050 con 0x7fa5ec15b2a0 2024-03-16T20:01:47.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5e9ffb640 1 -- 172.21.15.32:0/3180362317 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40af980 con 0x7fa5ec15b2a0 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 -- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 msgr2=0x7fa5ec15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec15b6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009d90 tx=0x7fa5e401f150 comp rx=0 tx=0).stop 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 -- 172.21.15.32:0/3180362317 shutdown_connections 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec15b6d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 --2- 172.21.15.32:0/3180362317 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec151ef0 0x7fa5ec0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 -- 172.21.15.32:0/3180362317 >> 172.21.15.32:0/3180362317 conn(0x7fa5ec070f90 msgr2=0x7fa5ec0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 -- 172.21.15.32:0/3180362317 shutdown_connections 2024-03-16T20:01:47.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.028+0000 7fa5f143b640 1 -- 172.21.15.32:0/3180362317 wait complete. 2024-03-16T20:01:47.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 Processor -- start 2024-03-16T20:01:47.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 -- start start 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec0c97b0 0x7fa5ec144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec06d980 con 0x7fa5ec151ef0 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0cd110 con 0x7fa5ec15b2a0 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5f143b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0cc740 con 0x7fa5ec0c97b0 2024-03-16T20:01:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5ea7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:47.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5ea7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34388/0 (socket says 172.21.15.32:34388) 2024-03-16T20:01:47.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5ea7fc640 1 -- 172.21.15.32:0/2146525606 learned_addr learned my addr 172.21.15.32:0/2146525606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:47.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5eb7fe640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:47.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.032+0000 7fa5eaffd640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec0c97b0 0x7fa5ec144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:47.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.036+0000 7fa5ea7fc640 1 -- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec0c97b0 msgr2=0x7fa5ec144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:47.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.036+0000 7fa5ea7fc640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec0c97b0 0x7fa5ec144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.036+0000 7fa5ea7fc640 1 -- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 msgr2=0x7fa5ec1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:47.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.036+0000 7fa5ea7fc640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec1497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.036+0000 7fa5ea7fc640 1 -- 172.21.15.32:0/2146525606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4009a70 con 0x7fa5ec151ef0 2024-03-16T20:01:47.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.040+0000 7fa5ea7fc640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c450 tx=0x7fa5e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.044+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0005cf0 con 0x7fa5ec151ef0 2024-03-16T20:01:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.044+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e0005e70 con 0x7fa5ec151ef0 2024-03-16T20:01:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.044+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0b3240 con 0x7fa5ec151ef0 2024-03-16T20:01:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.044+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00072f0 con 0x7fa5ec151ef0 2024-03-16T20:01:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.044+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec0cbe10 con 0x7fa5ec151ef0 2024-03-16T20:01:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.048+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec068000 con 0x7fa5ec151ef0 2024-03-16T20:01:47.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.328+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa5e00074b0 con 0x7fa5ec151ef0 2024-03-16T20:01:47.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.328+0000 7fa5d37fe640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 0x7fa5d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:47.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.332+0000 7fa5eaffd640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 0x7fa5d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:47.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.332+0000 7fa5eaffd640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 0x7fa5d8069300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc009b60 tx=0x7fa5dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.356+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa5e0055120 con 0x7fa5ec151ef0 2024-03-16T20:01:47.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.356+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa5e00513c0 con 0x7fa5ec151ef0 2024-03-16T20:01:47.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.636+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5ec151690 con 0x7fa5ec151ef0 2024-03-16T20:01:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5d37fe640 1 -- 172.21.15.32:0/2146525606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa5e0017070 con 0x7fa5ec151ef0 2024-03-16T20:01:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:47.683 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 msgr2=0x7fa5d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 0x7fa5d8069300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc009b60 tx=0x7fa5dc009190 comp rx=0 tx=0).stop 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 msgr2=0x7fa5ec0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c450 tx=0x7fa5e000c910 comp rx=0 tx=0).stop 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 shutdown_connections 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa5d8066e40 0x7fa5d8069300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec15b2a0 0x7fa5ec1497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ec151ef0 0x7fa5ec0cb7b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 --2- 172.21.15.32:0/2146525606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa5ec0c97b0 0x7fa5ec144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 >> 172.21.15.32:0/2146525606 conn(0x7fa5ec070f90 msgr2=0x7fa5ec0c32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 shutdown_connections 2024-03-16T20:01:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:47.708+0000 7fa5f143b640 1 -- 172.21.15.32:0/2146525606 wait complete. 2024-03-16T20:01:48.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:48.306 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:49.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:49.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 Processor -- start 2024-03-16T20:01:49.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 -- start start 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c4154350 0x7f41c406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41c40c2d70 0x7f41c415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f41c4072a10 0x7f41c4072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c406bae0 con 0x7f41c40c2d70 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c415a540 con 0x7f41c4072a10 2024-03-16T20:01:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41ca84e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c40c7fd0 con 0x7f41c4154350 2024-03-16T20:01:49.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41c37fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41c40c2d70 0x7f41c415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:49.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41c37fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41c40c2d70 0x7f41c415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35148/0 (socket says 172.21.15.32:35148) 2024-03-16T20:01:49.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41c37fe640 1 -- 172.21.15.32:0/786662764 learned_addr learned my addr 172.21.15.32:0/786662764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:49.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.468+0000 7f41c8deb640 1 --2- 172.21.15.32:0/786662764 >> v2:172.21.15.59:3300/0 conn(0x7f41c4072a10 0x7f41c4072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:49.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.472+0000 7f41c37fe640 1 -- 172.21.15.32:0/786662764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c4154350 msgr2=0x7f41c406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:49.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.472+0000 7f41c37fe640 1 --2- 172.21.15.32:0/786662764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c4154350 0x7f41c406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.472+0000 7f41c37fe640 1 -- 172.21.15.32:0/786662764 >> v2:172.21.15.59:3300/0 conn(0x7f41c4072a10 msgr2=0x7f41c4072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:49.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.472+0000 7f41c37fe640 1 --2- 172.21.15.32:0/786662764 >> v2:172.21.15.59:3300/0 conn(0x7f41c4072a10 0x7f41c4072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.472+0000 7f41c37fe640 1 -- 172.21.15.32:0/786662764 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c406bd10 con 0x7f41c40c2d70 2024-03-16T20:01:49.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.476+0000 7f41c37fe640 1 --2- 172.21.15.32:0/786662764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c40c2d70 0x7f41c415a9d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41c406ca80 tx=0x7f41b801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5324d4bbe20347 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41c2ffd640 1 -- 172.21.15.32:0/786662764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80ab600 con 0x7f41c40c2d70 2024-03-16T20:01:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41c2ffd640 1 -- 172.21.15.32:0/786662764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b80a7050 con 0x7f41c40c2d70 2024-03-16T20:01:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41c2ffd640 1 -- 172.21.15.32:0/786662764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80ab980 con 0x7f41c40c2d70 2024-03-16T20:01:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 -- 172.21.15.32:0/786662764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c40c2d70 msgr2=0x7f41c415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 --2- 172.21.15.32:0/786662764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c40c2d70 0x7f41c415a9d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41c406ca80 tx=0x7f41b801c470 comp rx=0 tx=0).stop 2024-03-16T20:01:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 -- 172.21.15.32:0/786662764 shutdown_connections 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 --2- 172.21.15.32:0/786662764 >> v2:172.21.15.59:3300/0 conn(0x7f41c4072a10 0x7f41c4072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 --2- 172.21.15.32:0/786662764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c40c2d70 0x7f41c415a9d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 --2- 172.21.15.32:0/786662764 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c4154350 0x7f41c406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 -- 172.21.15.32:0/786662764 >> 172.21.15.32:0/786662764 conn(0x7f41c406afc0 msgr2=0x7f41c406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 -- 172.21.15.32:0/786662764 shutdown_connections 2024-03-16T20:01:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.484+0000 7f41ca84e640 1 -- 172.21.15.32:0/786662764 wait complete. 2024-03-16T20:01:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 Processor -- start 2024-03-16T20:01:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 -- start start 2024-03-16T20:01:49.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c4072a10 0x7f41c4152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c40c2d70 0x7f41c4146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c41f3970 con 0x7f41c4154350 2024-03-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c4151d20 con 0x7f41c4072a10 2024-03-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41ca84e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c415cea0 con 0x7f41c40c2d70 2024-03-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41c8deb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41c3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c4072a10 0x7f41c4152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41c8deb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35152/0 (socket says 172.21.15.32:35152) 2024-03-16T20:01:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.488+0000 7f41c8deb640 1 -- 172.21.15.32:0/1059342006 learned_addr learned my addr 172.21.15.32:0/1059342006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.492+0000 7f41c8deb640 1 -- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c40c2d70 msgr2=0x7f41c4146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.492+0000 7f41c8deb640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c40c2d70 0x7f41c4146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.492+0000 7f41c8deb640 1 -- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c4072a10 msgr2=0x7f41c4152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.492+0000 7f41c8deb640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c4072a10 0x7f41c4152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.492+0000 7f41c8deb640 1 -- 172.21.15.32:0/1059342006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80094d0 con 0x7f41c4154350 2024-03-16T20:01:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.496+0000 7f41c8deb640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00efc0 tx=0x7f41bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc014480 con 0x7f41c4154350 2024-03-16T20:01:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc00f040 con 0x7f41c4154350 2024-03-16T20:01:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc014820 con 0x7f41c4154350 2024-03-16T20:01:49.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c4151fe0 con 0x7f41c4154350 2024-03-16T20:01:49.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c41525a0 con 0x7f41c4154350 2024-03-16T20:01:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.500+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c406edd0 con 0x7f41c4154350 2024-03-16T20:01:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.528+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f41bc014980 con 0x7f41c4154350 2024-03-16T20:01:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.528+0000 7f41c0ff9640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 0x7f41b0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.528+0000 7f41c3fff640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 0x7f41b0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:49.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.528+0000 7f41c3fff640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 0x7f41b0069380 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41b40098f0 tx=0x7f41b4006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:49.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.832+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f41bc056d90 con 0x7f41c4154350 2024-03-16T20:01:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:49.832+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f41bc052930 con 0x7f41c4154350 2024-03-16T20:01:50.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.092+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41c406edd0 con 0x7f41c4154350 2024-03-16T20:01:50.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.172+0000 7f41c0ff9640 1 -- 172.21.15.32:0/1059342006 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f41c406edd0 con 0x7f41c4154350 2024-03-16T20:01:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:50.148 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:50.148 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 msgr2=0x7f41b0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 0x7f41b0069380 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41b40098f0 tx=0x7f41b4006f90 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 msgr2=0x7f41c41f3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00efc0 tx=0x7f41bc00c490 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 shutdown_connections 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f41b0066ec0 0x7f41b0069380 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c4154350 0x7f41c41f3430 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41c40c2d70 0x7f41c4146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 --2- 172.21.15.32:0/1059342006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c4072a10 0x7f41c4152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 >> 172.21.15.32:0/1059342006 conn(0x7f41c406afc0 msgr2=0x7f41c40c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:50.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 shutdown_connections 2024-03-16T20:01:50.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:50.176+0000 7f41ca84e640 1 -- 172.21.15.32:0/1059342006 wait complete. 2024-03-16T20:01:50.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:50.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:51.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.908+0000 7f48bdad6640 1 Processor -- start 2024-03-16T20:01:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.908+0000 7f48bdad6640 1 -- start start 2024-03-16T20:01:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48bdad6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f48b8150c40 0x7f48b8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48bdad6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48b815a420 0x7f48b815a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48bdad6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b8151020 con 0x7f48b80725e0 2024-03-16T20:01:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48bdad6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b815a000 con 0x7f48b815a420 2024-03-16T20:01:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48bdad6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b815add0 con 0x7f48b8150c40 2024-03-16T20:01:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48b7fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48b815a420 0x7f48b815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48b7fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48b815a420 0x7f48b815a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35166/0 (socket says 172.21.15.32:35166) 2024-03-16T20:01:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48b7fff640 1 -- 172.21.15.32:0/962355700 learned_addr learned my addr 172.21.15.32:0/962355700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.912+0000 7f48b77fe640 1 --2- 172.21.15.32:0/962355700 >> v2:172.21.15.59:3300/0 conn(0x7f48b8150c40 0x7f48b8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.916+0000 7f48b7fff640 1 -- 172.21.15.32:0/962355700 >> v2:172.21.15.59:3300/0 conn(0x7f48b8150c40 msgr2=0x7f48b8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.916+0000 7f48b7fff640 1 --2- 172.21.15.32:0/962355700 >> v2:172.21.15.59:3300/0 conn(0x7f48b8150c40 0x7f48b8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.916+0000 7f48b7fff640 1 -- 172.21.15.32:0/962355700 >> v1:172.21.15.59:6789/0 conn(0x7f48b80725e0 legacy=0x7f48b8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.916+0000 7f48b7fff640 1 -- 172.21.15.32:0/962355700 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b8072da0 con 0x7f48b815a420 2024-03-16T20:01:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.920+0000 7f48b7fff640 1 --2- 172.21.15.32:0/962355700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b815a890 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f48b806cf60 tx=0x7f48b001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb9f62094b4cd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:51.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48b67fc640 1 -- 172.21.15.32:0/962355700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b00a58b0 con 0x7f48b815a420 2024-03-16T20:01:51.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48b67fc640 1 -- 172.21.15.32:0/962355700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b00a5a10 con 0x7f48b815a420 2024-03-16T20:01:51.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48b67fc640 1 -- 172.21.15.32:0/962355700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b00a5da0 con 0x7f48b815a420 2024-03-16T20:01:51.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 -- 172.21.15.32:0/962355700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 msgr2=0x7f48b815a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 --2- 172.21.15.32:0/962355700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b815a890 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f48b806cf60 tx=0x7f48b001c9f0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 -- 172.21.15.32:0/962355700 shutdown_connections 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 --2- 172.21.15.32:0/962355700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b815a890 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 --2- 172.21.15.32:0/962355700 >> v2:172.21.15.59:3300/0 conn(0x7f48b8150c40 0x7f48b8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 -- 172.21.15.32:0/962355700 >> 172.21.15.32:0/962355700 conn(0x7f48b806c0d0 msgr2=0x7f48b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 -- 172.21.15.32:0/962355700 shutdown_connections 2024-03-16T20:01:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.924+0000 7f48bdad6640 1 -- 172.21.15.32:0/962355700 wait complete. 2024-03-16T20:01:51.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 Processor -- start 2024-03-16T20:01:51.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 -- start start 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48b80725e0 0x7f48b81fb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48b8150c40 0x7f48b814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b8148380 con 0x7f48b815a420 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b81498f0 con 0x7f48b80725e0 2024-03-16T20:01:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48bdad6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b815d210 con 0x7f48b8150c40 2024-03-16T20:01:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48b7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48b80725e0 0x7f48b81fb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48b7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35174/0 (socket says 172.21.15.32:35174) 2024-03-16T20:01:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48b7fff640 1 -- 172.21.15.32:0/2167071749 learned_addr learned my addr 172.21.15.32:0/2167071749 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.928+0000 7f48b6ffd640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48b8150c40 0x7f48b814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.932+0000 7f48b7fff640 1 -- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48b8150c40 msgr2=0x7f48b814d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.932+0000 7f48b7fff640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48b8150c40 0x7f48b814d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.932+0000 7f48b7fff640 1 -- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48b80725e0 msgr2=0x7f48b81fb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.932+0000 7f48b7fff640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48b80725e0 0x7f48b81fb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.932+0000 7f48b7fff640 1 -- 172.21.15.32:0/2167071749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0009b30 con 0x7f48b815a420 2024-03-16T20:01:51.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.936+0000 7f48b7fff640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f48b00af040 tx=0x7f48b0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:51.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.940+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b00b8880 con 0x7f48b815a420 2024-03-16T20:01:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.940+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b00b8a00 con 0x7f48b815a420 2024-03-16T20:01:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.940+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b8149b40 con 0x7f48b815a420 2024-03-16T20:01:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.944+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b00b8db0 con 0x7f48b815a420 2024-03-16T20:01:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.944+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b8147d30 con 0x7f48b815a420 2024-03-16T20:01:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:51.944+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8068000 con 0x7f48b815a420 2024-03-16T20:01:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.260+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f48b00c3460 con 0x7f48b815a420 2024-03-16T20:01:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.260+0000 7f489ffff640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 0x7f48a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.260+0000 7f48b77fe640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 0x7f48a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.260+0000 7f48b77fe640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 0x7f48a4069350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48a80079c0 tx=0x7f48a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.280+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f48b00f1480 con 0x7f48b815a420 2024-03-16T20:01:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.280+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f48b00ed720 con 0x7f48b815a420 2024-03-16T20:01:52.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.496+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48b815d350 con 0x7f48b815a420 2024-03-16T20:01:52.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f489ffff640 1 -- 172.21.15.32:0/2167071749 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f48b00f0de0 con 0x7f48b815a420 2024-03-16T20:01:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:52.484 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:52.484 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:52.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 msgr2=0x7f48a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:52.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 0x7f48a4069350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48a80079c0 tx=0x7f48a8008040 comp rx=0 tx=0).stop 2024-03-16T20:01:52.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 msgr2=0x7f48b81f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:52.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f48b00af040 tx=0x7f48b0019790 comp rx=0 tx=0).stop 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 shutdown_connections 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48a4066e90 0x7f48a4069350 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48b815a420 0x7f48b81f72b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48b8150c40 0x7f48b814d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 --2- 172.21.15.32:0/2167071749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48b80725e0 0x7f48b81fb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 >> 172.21.15.32:0/2167071749 conn(0x7f48b806c0d0 msgr2=0x7f48b814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.508+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 shutdown_connections 2024-03-16T20:01:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:52.512+0000 7f48bdad6640 1 -- 172.21.15.32:0/2167071749 wait complete. 2024-03-16T20:01:53.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:53.111 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:54.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:54.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 Processor -- start 2024-03-16T20:01:54.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 -- start start 2024-03-16T20:01:54.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f78180b1eb0 0x7f7818072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78180c68e0 con 0x7f7818158980 2024-03-16T20:01:54.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7818067340 con 0x7f78180c7ce0 2024-03-16T20:01:54.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781f985640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781806c1f0 con 0x7f78180b1eb0 2024-03-16T20:01:54.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781df22640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7818158980 0x7f7818162730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33164/0 (socket says 172.21.15.32:33164) 2024-03-16T20:01:54.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.244+0000 7f781df22640 1 -- 172.21.15.32:0/184577681 learned_addr learned my addr 172.21.15.32:0/184577681 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:54.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.248+0000 7f781d721640 1 --2- 172.21.15.32:0/184577681 >> v2:172.21.15.59:3300/0 conn(0x7f78180b1eb0 0x7f7818072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.248+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942849874 0 0) 0x7f78180c68e0 con 0x7f7818158980 2024-03-16T20:01:54.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.248+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7804003530 con 0x7f7818158980 2024-03-16T20:01:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1630500416 0 0) 0x7f7804003530 con 0x7f7818158980 2024-03-16T20:01:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 >> v2:172.21.15.59:3300/0 conn(0x7f78180b1eb0 msgr2=0x7f7818072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 --2- 172.21.15.32:0/184577681 >> v2:172.21.15.59:3300/0 conn(0x7f78180b1eb0 0x7f7818072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 >> v1:172.21.15.59:6789/0 conn(0x7f78180c7ce0 legacy=0x7f7818158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781806c4d0 con 0x7f7818158980 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f7810001a00 con 0x7f7818158980 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 >> v1:172.21.15.32:6789/0 conn(0x7f7818158980 legacy=0x7f7818162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040039d0 0x7f7804003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 0x7f7804004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7804005190 0x7f780400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781806c1f0 con 0x7f78040047d0 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780400c070 con 0x7f78040039d0 2024-03-16T20:01:54.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.252+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780400c230 con 0x7f7804005190 2024-03-16T20:01:54.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781df22640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7804005190 0x7f780400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 0x7f7804004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 -- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7804005190 msgr2=0x7f780400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7804005190 0x7f780400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 -- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040039d0 msgr2=0x7f7804003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040039d0 0x7f7804003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781cf20640 1 -- 172.21.15.32:0/184577681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804003530 con 0x7f78040047d0 2024-03-16T20:01:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.256+0000 7f781df22640 1 -- 172.21.15.32:0/184577681 reap_dead start 2024-03-16T20:01:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.260+0000 7f781cf20640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 0x7f7804004c50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7808001750 tx=0x7f780801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8345860b05b2b260 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78080052d0 con 0x7f78040047d0 2024-03-16T20:01:54.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78080a7050 con 0x7f78040047d0 2024-03-16T20:01:54.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f780ffff640 1 -- 172.21.15.32:0/184577681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808003600 con 0x7f78040047d0 2024-03-16T20:01:54.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 msgr2=0x7f7804004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 0x7f7804004c50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7808001750 tx=0x7f780801c3c0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- 172.21.15.32:0/184577681 shutdown_connections 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 --2- 172.21.15.32:0/184577681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040047d0 0x7f7804004c50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- 172.21.15.32:0/184577681 >> 172.21.15.32:0/184577681 conn(0x7f781806afc0 msgr2=0x7f781806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- 172.21.15.32:0/184577681 shutdown_connections 2024-03-16T20:01:54.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- 172.21.15.32:0/184577681 wait complete. 2024-03-16T20:01:54.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 Processor -- start 2024-03-16T20:01:54.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.268+0000 7f781f985640 1 -- start start 2024-03-16T20:01:54.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040047d0 0x7f78181e3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78181edc30 0x7f78181ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181ed7e0 con 0x7f781814d840 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781814ddd0 con 0x7f78040047d0 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781f985640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78181ee5f0 con 0x7f78181edc30 2024-03-16T20:01:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781cf20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781df22640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78181edc30 0x7f78181ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781cf20640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35194/0 (socket says 172.21.15.32:35194) 2024-03-16T20:01:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781cf20640 1 -- 172.21.15.32:0/3411031099 learned_addr learned my addr 172.21.15.32:0/3411031099 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.272+0000 7f781d721640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040047d0 0x7f78181e3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.276+0000 7f781cf20640 1 -- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78181edc30 msgr2=0x7f78181ee0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.276+0000 7f781cf20640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78181edc30 0x7f78181ee0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.276+0000 7f781cf20640 1 -- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040047d0 msgr2=0x7f78181e3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.276+0000 7f781cf20640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040047d0 0x7f78181e3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.276+0000 7f781cf20640 1 -- 172.21.15.32:0/3411031099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781815c1b0 con 0x7f781814d840 2024-03-16T20:01:54.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.280+0000 7f781cf20640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7808005040 tx=0x7f7804005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:54.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808007440 con 0x7f781814d840 2024-03-16T20:01:54.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78080075a0 con 0x7f781814d840 2024-03-16T20:01:54.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781814c870 con 0x7f781814d840 2024-03-16T20:01:54.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808006a00 con 0x7f781814d840 2024-03-16T20:01:54.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781814ce30 con 0x7f781814d840 2024-03-16T20:01:54.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.288+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78181ed950 con 0x7f781814d840 2024-03-16T20:01:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.324+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7808006b60 con 0x7f781814d840 2024-03-16T20:01:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.328+0000 7f780dffb640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 0x7f77fc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.328+0000 7f781d721640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 0x7f77fc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:54.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.328+0000 7f781d721640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 0x7f77fc069330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78180693e0 tx=0x7f7814009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:54.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.604+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f78080f0750 con 0x7f781814d840 2024-03-16T20:01:54.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.604+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f78080ec2f0 con 0x7f781814d840 2024-03-16T20:01:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.812+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78181ed950 con 0x7f781814d840 2024-03-16T20:01:54.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.832+0000 7f780dffb640 1 -- 172.21.15.32:0/3411031099 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f78181ed950 con 0x7f781814d840 2024-03-16T20:01:54.810 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:54.810 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:54.810 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 msgr2=0x7f77fc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 0x7f77fc069330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78180693e0 tx=0x7f7814009190 comp rx=0 tx=0).stop 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 msgr2=0x7f781815b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7808005040 tx=0x7f7804005800 comp rx=0 tx=0).stop 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 shutdown_connections 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77fc066e70 0x7f77fc069330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78181edc30 0x7f78181ee0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f781814d840 0x7f781815b630 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 --2- 172.21.15.32:0/3411031099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78040047d0 0x7f78181e3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:54.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 >> 172.21.15.32:0/3411031099 conn(0x7f781806afc0 msgr2=0x7f7818072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:54.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 shutdown_connections 2024-03-16T20:01:54.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:54.836+0000 7f781f985640 1 -- 172.21.15.32:0/3411031099 wait complete. 2024-03-16T20:01:55.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:55.435 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:56.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:56.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 Processor -- start 2024-03-16T20:01:56.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 -- start start 2024-03-16T20:01:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca74151b40 0x7fca7406f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca74151f20 0x7fca740ca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca74158a70 con 0x7fca7406f880 2024-03-16T20:01:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7406fe40 con 0x7fca74151f20 2024-03-16T20:01:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.567+0000 7fca79369640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca74157c50 con 0x7fca74151b40 2024-03-16T20:01:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca74151f20 0x7fca740ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca74151f20 0x7fca740ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35202/0 (socket says 172.21.15.32:35202) 2024-03-16T20:01:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 -- 172.21.15.32:0/81482780 learned_addr learned my addr 172.21.15.32:0/81482780 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca72ffd640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca74151b40 0x7fca7406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 -- 172.21.15.32:0/81482780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca74151b40 msgr2=0x7fca7406f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca74151b40 0x7fca7406f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 -- 172.21.15.32:0/81482780 >> v1:172.21.15.59:6789/0 conn(0x7fca7406f880 legacy=0x7fca7415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.571+0000 7fca727fc640 1 -- 172.21.15.32:0/81482780 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7406e9f0 con 0x7fca74151f20 2024-03-16T20:01:56.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.575+0000 7fca727fc640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca74151f20 0x7fca740ca020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fca68009870 tx=0x7fca6801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ac80ec1fd75542 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca71ffb640 1 -- 172.21.15.32:0/81482780 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680a8a50 con 0x7fca74151f20 2024-03-16T20:01:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca71ffb640 1 -- 172.21.15.32:0/81482780 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680a8bb0 con 0x7fca74151f20 2024-03-16T20:01:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca71ffb640 1 -- 172.21.15.32:0/81482780 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680b0440 con 0x7fca74151f20 2024-03-16T20:01:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 -- 172.21.15.32:0/81482780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca74151f20 msgr2=0x7fca740ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca74151f20 0x7fca740ca020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fca68009870 tx=0x7fca6801ed50 comp rx=0 tx=0).stop 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 -- 172.21.15.32:0/81482780 shutdown_connections 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca74151f20 0x7fca740ca020 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 --2- 172.21.15.32:0/81482780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca74151b40 0x7fca7406f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 -- 172.21.15.32:0/81482780 >> 172.21.15.32:0/81482780 conn(0x7fca74066cd0 msgr2=0x7fca740670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 -- 172.21.15.32:0/81482780 shutdown_connections 2024-03-16T20:01:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 -- 172.21.15.32:0/81482780 wait complete. 2024-03-16T20:01:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.583+0000 7fca79369640 1 Processor -- start 2024-03-16T20:01:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 -- start start 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca7406f880 0x7fca7414e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca74151b40 0x7fca7414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740cde70 con 0x7fca7414dd40 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740ce3c0 con 0x7fca74151b40 2024-03-16T20:01:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca79369640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca741f1b20 con 0x7fca7406f880 2024-03-16T20:01:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35214/0 (socket says 172.21.15.32:35214) 2024-03-16T20:01:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca737fe640 1 -- 172.21.15.32:0/2004520012 learned_addr learned my addr 172.21.15.32:0/2004520012 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca72ffd640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca7406f880 0x7fca7414e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.587+0000 7fca727fc640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca74151b40 0x7fca7414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.591+0000 7fca737fe640 1 -- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca7406f880 msgr2=0x7fca7414e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.591+0000 7fca737fe640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca7406f880 0x7fca7414e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.591+0000 7fca737fe640 1 -- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca74151b40 msgr2=0x7fca7414c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.591+0000 7fca737fe640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca74151b40 0x7fca7414c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.591+0000 7fca737fe640 1 -- 172.21.15.32:0/2004520012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca680094d0 con 0x7fca7414dd40 2024-03-16T20:01:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.595+0000 7fca737fe640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c770 tx=0x7fca6c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.599+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c003b80 con 0x7fca7414dd40 2024-03-16T20:01:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.599+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c003ce0 con 0x7fca7414dd40 2024-03-16T20:01:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.599+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c00aa90 con 0x7fca7414dd40 2024-03-16T20:01:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.599+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca741f1e00 con 0x7fca7414dd40 2024-03-16T20:01:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.599+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca740cb4c0 con 0x7fca7414dd40 2024-03-16T20:01:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.603+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca740cb6e0 con 0x7fca7414dd40 2024-03-16T20:01:56.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.667+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fca6c00abf0 con 0x7fca7414dd40 2024-03-16T20:01:56.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.667+0000 7fca5b7fe640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 0x7fca60069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:56.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.667+0000 7fca72ffd640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 0x7fca60069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:56.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.667+0000 7fca72ffd640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 0x7fca60069350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca64009b60 tx=0x7fca64009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:56.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.671+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fca6c055ec0 con 0x7fca7414dd40 2024-03-16T20:01:56.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:56.919+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fca6c052160 con 0x7fca7414dd40 2024-03-16T20:01:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.175+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7406a340 con 0x7fca7414dd40 2024-03-16T20:01:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.187+0000 7fca5b7fe640 1 -- 172.21.15.32:0/2004520012 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fca6c055820 con 0x7fca7414dd40 2024-03-16T20:01:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:57.167 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 msgr2=0x7fca60069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 0x7fca60069350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca64009b60 tx=0x7fca64009190 comp rx=0 tx=0).stop 2024-03-16T20:01:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 msgr2=0x7fca740cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c770 tx=0x7fca6c00cc30 comp rx=0 tx=0).stop 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 shutdown_connections 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fca60066e90 0x7fca60069350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca7414dd40 0x7fca740cbfe0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca74151b40 0x7fca7414c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 --2- 172.21.15.32:0/2004520012 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca7406f880 0x7fca7414e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 >> 172.21.15.32:0/2004520012 conn(0x7fca74066cd0 msgr2=0x7fca740ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 shutdown_connections 2024-03-16T20:01:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:57.191+0000 7fca79369640 1 -- 172.21.15.32:0/2004520012 wait complete. 2024-03-16T20:01:57.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:01:57.791 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:01:58.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:01:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 Processor -- start 2024-03-16T20:01:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 -- start start 2024-03-16T20:01:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7250150d70 0x7f7250073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f725015a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7250072dd0 con 0x7f72500725e0 2024-03-16T20:01:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725015d790 con 0x7f7250150d70 2024-03-16T20:01:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7257219640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725015da50 con 0x7f725015a420 2024-03-16T20:01:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f724ffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f72500725e0 0x7f7250070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33166/0 (socket says 172.21.15.32:33166) 2024-03-16T20:01:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f724ffff640 1 -- 172.21.15.32:0/2350154767 learned_addr learned my addr 172.21.15.32:0/2350154767 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:01:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.923+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2350154767 >> v2:172.21.15.59:3300/0 conn(0x7f7250150d70 0x7f7250073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.927+0000 7f72557b6640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f725015a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.927+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933427642 0 0) 0x7f7250072dd0 con 0x7f72500725e0 2024-03-16T20:01:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.927+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7240003530 con 0x7f72500725e0 2024-03-16T20:01:58.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565982754 0 0) 0x7f7240003530 con 0x7f72500725e0 2024-03-16T20:01:58.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 msgr2=0x7f725015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:58.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f725015a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 >> v2:172.21.15.59:3300/0 conn(0x7f7250150d70 msgr2=0x7f7250073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 --2- 172.21.15.32:0/2350154767 >> v2:172.21.15.59:3300/0 conn(0x7f7250150d70 0x7f7250073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725015dd30 con 0x7f72500725e0 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f723c001680 con 0x7f72500725e0 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 >> v1:172.21.15.32:6789/0 conn(0x7f72500725e0 legacy=0x7f7250070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f7240003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f72400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7240004d10 0x7f7240005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725015d790 con 0x7f7240003980 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725015da50 con 0x7f7240004d10 2024-03-16T20:01:58.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.931+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724000ba60 con 0x7f725015a420 2024-03-16T20:01:58.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.935+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f7240003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.935+0000 7f72557b6640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7240004d10 0x7f7240005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f724ffff640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f72400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f7254fb5640 1 -- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 msgr2=0x7f72400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015a420 0x7f72400047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f7254fb5640 1 -- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7240004d10 msgr2=0x7f7240005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:01:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7240004d10 0x7f7240005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.939+0000 7f7254fb5640 1 -- 172.21.15.32:0/2350154767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240003530 con 0x7f7240003980 2024-03-16T20:01:58.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.943+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f7240003da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7244009ae0 tx=0x7f724401ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a27b61c81c8e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72440a5650 con 0x7f7240003980 2024-03-16T20:01:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72440a57b0 con 0x7f7240003980 2024-03-16T20:01:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f724f7fe640 1 -- 172.21.15.32:0/2350154767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72440a5b60 con 0x7f7240003980 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 -- 172.21.15.32:0/2350154767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 msgr2=0x7f7240003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 --2- 172.21.15.32:0/2350154767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f7240003da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7244009ae0 tx=0x7f724401ca80 comp rx=0 tx=0).stop 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f72557b6640 1 -- 172.21.15.32:0/2350154767 reap_dead start 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 -- 172.21.15.32:0/2350154767 shutdown_connections 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 -- 172.21.15.32:0/2350154767 >> 172.21.15.32:0/2350154767 conn(0x7f725006c0d0 msgr2=0x7f725006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 -- 172.21.15.32:0/2350154767 shutdown_connections 2024-03-16T20:01:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.947+0000 7f7257219640 1 -- 172.21.15.32:0/2350154767 wait complete. 2024-03-16T20:01:58.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 Processor -- start 2024-03-16T20:01:58.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 -- start start 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015bb30 0x7f725013eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f725015bf50 0x7f725013f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500cc170 con 0x7f7240003980 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500cae90 con 0x7f725015bf50 2024-03-16T20:01:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7257219640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72501f3c30 con 0x7f725015bb30 2024-03-16T20:01:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7254fb5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f72557b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f725015bf50 0x7f725013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7254fb5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35232/0 (socket says 172.21.15.32:35232) 2024-03-16T20:01:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f7254fb5640 1 -- 172.21.15.32:0/2896426597 learned_addr learned my addr 172.21.15.32:0/2896426597 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:01:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.951+0000 7f724ffff640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015bb30 0x7f725013eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.955+0000 7f7254fb5640 1 -- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015bb30 msgr2=0x7f725013eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.955+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015bb30 0x7f725013eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.955+0000 7f7254fb5640 1 -- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f725015bf50 msgr2=0x7f725013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.955+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f725015bf50 0x7f725013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.955+0000 7f7254fb5640 1 -- 172.21.15.32:0/2896426597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725015d560 con 0x7f7240003980 2024-03-16T20:01:58.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.959+0000 7f7254fb5640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7244009ae0 tx=0x7f7240005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:58.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.963+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72440be070 con 0x7f7240003980 2024-03-16T20:01:58.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.963+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724400f000 con 0x7f7240003980 2024-03-16T20:01:58.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.963+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72501f3ed0 con 0x7f7240003980 2024-03-16T20:01:58.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.963+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72440a5650 con 0x7f7240003980 2024-03-16T20:01:58.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.963+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72500ca860 con 0x7f7240003980 2024-03-16T20:01:58.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:58.967+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725015d380 con 0x7f7240003980 2024-03-16T20:01:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.247+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f72440b4040 con 0x7f7240003980 2024-03-16T20:01:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.247+0000 7f724d7fa640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 0x7f7234069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:01:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.247+0000 7f724ffff640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 0x7f7234069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:01:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.247+0000 7f724ffff640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 0x7f7234069350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f723c000c10 tx=0x7f723c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:01:59.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.271+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f72440ef830 con 0x7f7240003980 2024-03-16T20:01:59.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.271+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f72440ebad0 con 0x7f7240003980 2024-03-16T20:01:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.547+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72500cac80 con 0x7f7240003980 2024-03-16T20:01:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f724d7fa640 1 -- 172.21.15.32:0/2896426597 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f72440b2090 con 0x7f7240003980 2024-03-16T20:01:59.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:01:59.555 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:01:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:01:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 msgr2=0x7f7234069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 0x7f7234069350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f723c000c10 tx=0x7f723c009040 comp rx=0 tx=0).stop 2024-03-16T20:01:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 msgr2=0x7f725015c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:01:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7244009ae0 tx=0x7f7240005800 comp rx=0 tx=0).stop 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 shutdown_connections 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7234066e90 0x7f7234069350 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f725015bf50 0x7f725013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f725015bb30 0x7f725013eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 --2- 172.21.15.32:0/2896426597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7240003980 0x7f725015c3f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 >> 172.21.15.32:0/2896426597 conn(0x7f725006c0d0 msgr2=0x7f725006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 shutdown_connections 2024-03-16T20:01:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:01:59.579+0000 7f7257219640 1 -- 172.21.15.32:0/2896426597 wait complete. 2024-03-16T20:02:00.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:02:00.179 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:01.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:01.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 Processor -- start 2024-03-16T20:02:01.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 -- start start 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 0x7f5e3c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5e3c0c8980 0x7f5e3c15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0c8ec0 con 0x7f5e3c151000 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c072f10 con 0x7f5e3c0c8980 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.311+0000 7f5e4094e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0c5000 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e3a59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5e3c151000 0x7f5e3c0c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49982/0 (socket says 172.21.15.32:49982) 2024-03-16T20:02:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e3a59c640 1 -- 172.21.15.32:0/3952266492 learned_addr learned my addr 172.21.15.32:0/3952266492 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e33fff640 1 --2- 172.21.15.32:0/3952266492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 0x7f5e3c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e3ad9d640 1 --2- 172.21.15.32:0/3952266492 >> v2:172.21.15.59:3300/0 conn(0x7f5e3c0c8980 0x7f5e3c15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e39d9b640 1 -- 172.21.15.32:0/3952266492 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873370748 0 0) 0x7f5e3c0c8ec0 con 0x7f5e3c151000 2024-03-16T20:02:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.315+0000 7f5e39d9b640 1 -- 172.21.15.32:0/3952266492 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e28005740 con 0x7f5e3c151000 2024-03-16T20:02:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.319+0000 7f5e33fff640 1 -- 172.21.15.32:0/3952266492 >> v2:172.21.15.59:3300/0 conn(0x7f5e3c0c8980 msgr2=0x7f5e3c15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.319+0000 7f5e33fff640 1 --2- 172.21.15.32:0/3952266492 >> v2:172.21.15.59:3300/0 conn(0x7f5e3c0c8980 0x7f5e3c15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.319+0000 7f5e33fff640 1 -- 172.21.15.32:0/3952266492 >> v1:172.21.15.32:6789/0 conn(0x7f5e3c151000 legacy=0x7f5e3c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.319+0000 7f5e33fff640 1 -- 172.21.15.32:0/3952266492 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c072950 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.327+0000 7f5e33fff640 1 --2- 172.21.15.32:0/3952266492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 0x7f5e3c0c8420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e240097c0 tx=0x7f5e2401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb39d18f94f6c566 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.331+0000 7f5e39d9b640 1 -- 172.21.15.32:0/3952266492 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240ac600 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.331+0000 7f5e39d9b640 1 -- 172.21.15.32:0/3952266492 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e240a8050 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.331+0000 7f5e39d9b640 1 -- 172.21.15.32:0/3952266492 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240ac980 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.331+0000 7f5e4094e640 1 -- 172.21.15.32:0/3952266492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 msgr2=0x7f5e3c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.331+0000 7f5e4094e640 1 --2- 172.21.15.32:0/3952266492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 0x7f5e3c0c8420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e240097c0 tx=0x7f5e2401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- 172.21.15.32:0/3952266492 shutdown_connections 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 --2- 172.21.15.32:0/3952266492 >> v2:172.21.15.59:3300/0 conn(0x7f5e3c0c8980 0x7f5e3c15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 --2- 172.21.15.32:0/3952266492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c36b0 0x7f5e3c0c8420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- 172.21.15.32:0/3952266492 >> 172.21.15.32:0/3952266492 conn(0x7f5e3c06c0d0 msgr2=0x7f5e3c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- 172.21.15.32:0/3952266492 shutdown_connections 2024-03-16T20:02:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- 172.21.15.32:0/3952266492 wait complete. 2024-03-16T20:02:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 Processor -- start 2024-03-16T20:02:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- start start 2024-03-16T20:02:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c8980 0x7f5e3c1e8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e3c151000 0x7f5e3c13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c159670 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c145590 con 0x7f5e3c151000 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e4094e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c158a70 con 0x7f5e3c0c8980 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.335+0000 7f5e3a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e3a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46434/0 (socket says 172.21.15.32:46434) 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e3a59c640 1 -- 172.21.15.32:0/4291979205 learned_addr learned my addr 172.21.15.32:0/4291979205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e33fff640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c8980 0x7f5e3c1e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e3ad9d640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e3c151000 0x7f5e3c13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e3a59c640 1 -- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c8980 msgr2=0x7f5e3c1e8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.339+0000 7f5e3a59c640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c8980 0x7f5e3c1e8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.343+0000 7f5e3a59c640 1 -- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e3c151000 msgr2=0x7f5e3c13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.343+0000 7f5e3a59c640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e3c151000 0x7f5e3c13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.343+0000 7f5e3a59c640 1 -- 172.21.15.32:0/4291979205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24009420 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.343+0000 7f5e3a59c640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c3f0 tx=0x7f5e2c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:01.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.347+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c005cf0 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.347+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c005e50 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.347+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0072f0 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.351+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1452b0 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.351+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c145db0 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.351+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c068000 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.643+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5e2c007450 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.643+0000 7f5e32ffd640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 0x7f5e1c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:01.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.643+0000 7f5e33fff640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 0x7f5e1c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:01.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.643+0000 7f5e33fff640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 0x7f5e1c069350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e240ae000 tx=0x7f5e240a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.675+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f5e2c055090 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.675+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5e2c051330 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.891+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c1613b0 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e32ffd640 1 -- 172.21.15.32:0/4291979205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f5e2c017020 con 0x7f5e3c0c36b0 2024-03-16T20:02:01.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:01.959 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:01:37.115676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:01:37.115673+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:02:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 msgr2=0x7f5e1c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 0x7f5e1c069350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e240ae000 tx=0x7f5e240a7000 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 msgr2=0x7f5e3c1e83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c3f0 tx=0x7f5e2c00c8b0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 shutdown_connections 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5e1c066e90 0x7f5e1c069350 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e3c151000 0x7f5e3c13ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e3c0c8980 0x7f5e3c1e8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 --2- 172.21.15.32:0/4291979205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3c0c36b0 0x7f5e3c1e83f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 >> 172.21.15.32:0/4291979205 conn(0x7f5e3c06c0d0 msgr2=0x7f5e3c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 shutdown_connections 2024-03-16T20:02:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:01.983+0000 7f5e4094e640 1 -- 172.21.15.32:0/4291979205 wait complete. 2024-03-16T20:02:02.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:01:37.115676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:01:37.115673+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:02:02.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:03.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:03.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 Processor -- start 2024-03-16T20:02:03.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 -- start start 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8de4153fc0 0x7f8de40c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8de40c72f0 0x7f8de415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de415ad50 con 0x7f8de415df30 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.711+0000 7f8de94f2640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de40c7ce0 con 0x7f8de4153fc0 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.715+0000 7f8de94f2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de415a1f0 con 0x7f8de40c72f0 2024-03-16T20:02:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.715+0000 7f8de2ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8de4153fc0 0x7f8de40c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.715+0000 7f8de2ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8de4153fc0 0x7f8de40c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46450/0 (socket says 172.21.15.32:46450) 2024-03-16T20:02:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.715+0000 7f8de2ffd640 1 -- 172.21.15.32:0/50194462 learned_addr learned my addr 172.21.15.32:0/50194462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.715+0000 7f8de27fc640 1 --2- 172.21.15.32:0/50194462 >> v2:172.21.15.59:3300/0 conn(0x7f8de40c72f0 0x7f8de415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.719+0000 7f8de2ffd640 1 -- 172.21.15.32:0/50194462 >> v2:172.21.15.59:3300/0 conn(0x7f8de40c72f0 msgr2=0x7f8de415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.719+0000 7f8de2ffd640 1 --2- 172.21.15.32:0/50194462 >> v2:172.21.15.59:3300/0 conn(0x7f8de40c72f0 0x7f8de415d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.719+0000 7f8de2ffd640 1 -- 172.21.15.32:0/50194462 >> v1:172.21.15.32:6789/0 conn(0x7f8de415df30 legacy=0x7f8de40c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.719+0000 7f8de2ffd640 1 -- 172.21.15.32:0/50194462 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de415a4b0 con 0x7f8de4153fc0 2024-03-16T20:02:03.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.723+0000 7f8de2ffd640 1 --2- 172.21.15.32:0/50194462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4153fc0 0x7f8de40c8b90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009870 tx=0x7f8dd401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6f3db4f8bf4c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:03.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.727+0000 7f8de1ffb640 1 -- 172.21.15.32:0/50194462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40a89d0 con 0x7f8de4153fc0 2024-03-16T20:02:03.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.727+0000 7f8de1ffb640 1 -- 172.21.15.32:0/50194462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd40a8b30 con 0x7f8de4153fc0 2024-03-16T20:02:03.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.727+0000 7f8de1ffb640 1 -- 172.21.15.32:0/50194462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40b0440 con 0x7f8de4153fc0 2024-03-16T20:02:03.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.727+0000 7f8de94f2640 1 -- 172.21.15.32:0/50194462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4153fc0 msgr2=0x7f8de40c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:03.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.727+0000 7f8de94f2640 1 --2- 172.21.15.32:0/50194462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4153fc0 0x7f8de40c8b90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009870 tx=0x7f8dd401ecd0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- 172.21.15.32:0/50194462 shutdown_connections 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 --2- 172.21.15.32:0/50194462 >> v2:172.21.15.59:3300/0 conn(0x7f8de40c72f0 0x7f8de415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 --2- 172.21.15.32:0/50194462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4153fc0 0x7f8de40c8b90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- 172.21.15.32:0/50194462 >> 172.21.15.32:0/50194462 conn(0x7f8de40cc7a0 msgr2=0x7f8de40ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- 172.21.15.32:0/50194462 shutdown_connections 2024-03-16T20:02:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- 172.21.15.32:0/50194462 wait complete. 2024-03-16T20:02:03.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 Processor -- start 2024-03-16T20:02:03.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- start start 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de40c72f0 0x7f8de4152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8de4153fc0 0x7f8de41eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4151a90 con 0x7f8de415df30 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4150630 con 0x7f8de40c72f0 2024-03-16T20:02:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de94f2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de41516c0 con 0x7f8de4153fc0 2024-03-16T20:02:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de40c72f0 0x7f8de4152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46466/0 (socket says 172.21.15.32:46466) 2024-03-16T20:02:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de37fe640 1 -- 172.21.15.32:0/3981197905 learned_addr learned my addr 172.21.15.32:0/3981197905 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.731+0000 7f8de27fc640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8de4153fc0 0x7f8de41eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.735+0000 7f8de37fe640 1 -- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8de4153fc0 msgr2=0x7f8de41eff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.735+0000 7f8de37fe640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8de4153fc0 0x7f8de41eff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.735+0000 7f8de37fe640 1 -- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de40c72f0 msgr2=0x7f8de4152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.735+0000 7f8de37fe640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de40c72f0 0x7f8de4152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.735+0000 7f8de37fe640 1 -- 172.21.15.32:0/3981197905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40094d0 con 0x7f8de415df30 2024-03-16T20:02:03.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.739+0000 7f8de37fe640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00c780 tx=0x7f8ddc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc003d90 con 0x7f8de415df30 2024-03-16T20:02:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ddc00ce90 con 0x7f8de415df30 2024-03-16T20:02:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc0052b0 con 0x7f8de415df30 2024-03-16T20:02:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de4150860 con 0x7f8de415df30 2024-03-16T20:02:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de4150e20 con 0x7f8de415df30 2024-03-16T20:02:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.743+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de41f1450 con 0x7f8de415df30 2024-03-16T20:02:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.771+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8ddc006b20 con 0x7f8de415df30 2024-03-16T20:02:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.771+0000 7f8dcb7fe640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 0x7f8dd0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.771+0000 7f8de2ffd640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 0x7f8dd0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:03.771+0000 7f8de2ffd640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 0x7f8dd0069330 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40b2000 tx=0x7f8dd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:04.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.047+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8ddc054ba0 con 0x7f8de415df30 2024-03-16T20:02:04.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.047+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8ddc050e40 con 0x7f8de415df30 2024-03-16T20:02:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.267+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8de4068000 con 0x7f8de415df30 2024-03-16T20:02:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.291+0000 7f8dcb7fe640 1 -- 172.21.15.32:0/3981197905 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8ddc054500 con 0x7f8de415df30 2024-03-16T20:02:04.269 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:04.269 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 msgr2=0x7f8dd0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 0x7f8dd0069330 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40b2000 tx=0x7f8dd4009040 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 msgr2=0x7f8de4158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00c780 tx=0x7f8ddc00cc40 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 shutdown_connections 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dd0066e70 0x7f8dd0069330 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de415df30 0x7f8de4158d20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8de4153fc0 0x7f8de41eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 --2- 172.21.15.32:0/3981197905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de40c72f0 0x7f8de4152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 >> 172.21.15.32:0/3981197905 conn(0x7f8de40cc7a0 msgr2=0x7f8de406bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 shutdown_connections 2024-03-16T20:02:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:04.295+0000 7f8de94f2640 1 -- 172.21.15.32:0/3981197905 wait complete. 2024-03-16T20:02:04.892 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:04.892 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:05.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 Processor -- start 2024-03-16T20:02:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 -- start start 2024-03-16T20:02:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93300c6040 0x7f93300c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f9330157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9330070760 con 0x7f933006ba30 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93300731c0 con 0x7f93300c6040 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.023+0000 7f9334f15640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93300734a0 con 0x7f93300c6420 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93300c6040 0x7f93300c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93300c6040 0x7f93300c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46468/0 (socket says 172.21.15.32:46468) 2024-03-16T20:02:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 -- 172.21.15.32:0/3438097885 learned_addr learned my addr 172.21.15.32:0/3438097885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932dd9b640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f9330157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 -- 172.21.15.32:0/3438097885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 msgr2=0x7f9330157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f9330157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 -- 172.21.15.32:0/3438097885 >> v1:172.21.15.32:6789/0 conn(0x7f933006ba30 legacy=0x7f93301516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.027+0000 7f932e59c640 1 -- 172.21.15.32:0/3438097885 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933006fe30 con 0x7f93300c6040 2024-03-16T20:02:06.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.031+0000 7f932e59c640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93300c8800 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9320009870 tx=0x7f932001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e653cef178e804 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:06.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f932d59a640 1 -- 172.21.15.32:0/3438097885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93200ac620 con 0x7f93300c6040 2024-03-16T20:02:06.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f932d59a640 1 -- 172.21.15.32:0/3438097885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93200ac780 con 0x7f93300c6040 2024-03-16T20:02:06.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f932d59a640 1 -- 172.21.15.32:0/3438097885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93200acb10 con 0x7f93300c6040 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 -- 172.21.15.32:0/3438097885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 msgr2=0x7f93300c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93300c8800 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9320009870 tx=0x7f932001c4f0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 -- 172.21.15.32:0/3438097885 shutdown_connections 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f9330157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 --2- 172.21.15.32:0/3438097885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93300c8800 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 -- 172.21.15.32:0/3438097885 >> 172.21.15.32:0/3438097885 conn(0x7f933006c0d0 msgr2=0x7f933006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 -- 172.21.15.32:0/3438097885 shutdown_connections 2024-03-16T20:02:06.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.039+0000 7f9334f15640 1 -- 172.21.15.32:0/3438097885 wait complete. 2024-03-16T20:02:06.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 Processor -- start 2024-03-16T20:02:06.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 -- start start 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f933006ba30 0x7f933015af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f93300c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933015b4d0 con 0x7f93300c6040 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933015b7a0 con 0x7f933006ba30 2024-03-16T20:02:06.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f9334f15640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93301f3ac0 con 0x7f93300c6420 2024-03-16T20:02:06.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f932dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f932dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46476/0 (socket says 172.21.15.32:46476) 2024-03-16T20:02:06.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f932dd9b640 1 -- 172.21.15.32:0/94064394 learned_addr learned my addr 172.21.15.32:0/94064394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:06.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f932e59c640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f933006ba30 0x7f933015af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.043+0000 7f932ed9d640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f93300c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.047+0000 7f932dd9b640 1 -- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 msgr2=0x7f93300c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.047+0000 7f932dd9b640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f93300c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.047+0000 7f932dd9b640 1 -- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f933006ba30 msgr2=0x7f933015af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.047+0000 7f932dd9b640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f933006ba30 0x7f933015af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.047+0000 7f932dd9b640 1 -- 172.21.15.32:0/94064394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200094d0 con 0x7f93300c6040 2024-03-16T20:02:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.051+0000 7f932dd9b640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f932400e870 tx=0x7f932400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.051+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240094d0 con 0x7f93300c6040 2024-03-16T20:02:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.055+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93301f3d60 con 0x7f93300c6040 2024-03-16T20:02:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.055+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9330159b10 con 0x7f93300c6040 2024-03-16T20:02:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.055+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93300729b0 con 0x7f93300c6040 2024-03-16T20:02:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.055+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93240059a0 con 0x7f93300c6040 2024-03-16T20:02:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.055+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324005d10 con 0x7f93300c6040 2024-03-16T20:02:06.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.363+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f932400c940 con 0x7f93300c6040 2024-03-16T20:02:06.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.363+0000 7f931effd640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 0x7f9310069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.363+0000 7f932e59c640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 0x7f9310069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.363+0000 7f932e59c640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 0x7f9310069300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93200ae000 tx=0x7f93200a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.379+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9324055ea0 con 0x7f93300c6040 2024-03-16T20:02:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.379+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9324052140 con 0x7f93300c6040 2024-03-16T20:02:06.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.611+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9330072e30 con 0x7f93300c6040 2024-03-16T20:02:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.623+0000 7f931effd640 1 -- 172.21.15.32:0/94064394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9324021090 con 0x7f93300c6040 2024-03-16T20:02:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:06.602 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 msgr2=0x7f9310069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 0x7f9310069300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93200ae000 tx=0x7f93200a7000 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 msgr2=0x7f93301f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f932400e870 tx=0x7f932400ed30 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 shutdown_connections 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9310066e40 0x7f9310069300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93300c6420 0x7f93300c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93300c6040 0x7f93301f3180 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 --2- 172.21.15.32:0/94064394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f933006ba30 0x7f933015af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 >> 172.21.15.32:0/94064394 conn(0x7f933006c0d0 msgr2=0x7f933014f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 shutdown_connections 2024-03-16T20:02:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:06.627+0000 7f9334f15640 1 -- 172.21.15.32:0/94064394 wait complete. 2024-03-16T20:02:07.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:07.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:08.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:08.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.355+0000 7f03304a8640 1 Processor -- start 2024-03-16T20:02:08.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.355+0000 7f03304a8640 1 -- start start 2024-03-16T20:02:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f03304a8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f03280725e0 0x7f0328073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f03304a8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f032815a420 0x7f032815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f03304a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328072dd0 con 0x7f0328150d70 2024-03-16T20:02:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f03304a8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032815d790 con 0x7f03280725e0 2024-03-16T20:02:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f03304a8640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032815da50 con 0x7f032815a420 2024-03-16T20:02:08.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f032da43640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f03280725e0 0x7f0328073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f032da43640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f03280725e0 0x7f0328073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46484/0 (socket says 172.21.15.32:46484) 2024-03-16T20:02:08.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f032da43640 1 -- 172.21.15.32:0/3096549268 learned_addr learned my addr 172.21.15.32:0/3096549268 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:08.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.359+0000 7f032ea45640 1 --2- 172.21.15.32:0/3096549268 >> v2:172.21.15.59:3300/0 conn(0x7f032815a420 0x7f032815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.363+0000 7f032da43640 1 -- 172.21.15.32:0/3096549268 >> v2:172.21.15.59:3300/0 conn(0x7f032815a420 msgr2=0x7f032815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:08.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.363+0000 7f032da43640 1 --2- 172.21.15.32:0/3096549268 >> v2:172.21.15.59:3300/0 conn(0x7f032815a420 0x7f032815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.363+0000 7f032da43640 1 -- 172.21.15.32:0/3096549268 >> v1:172.21.15.59:6789/0 conn(0x7f0328150d70 legacy=0x7f0328070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.363+0000 7f032da43640 1 -- 172.21.15.32:0/3096549268 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032815dd30 con 0x7f03280725e0 2024-03-16T20:02:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.367+0000 7f032da43640 1 --2- 172.21.15.32:0/3096549268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03280725e0 0x7f0328073070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f03280c4a30 tx=0x7f031801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bd1c66e6344537 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f032d242640 1 -- 172.21.15.32:0/3096549268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180ac630 con 0x7f03280725e0 2024-03-16T20:02:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f032d242640 1 -- 172.21.15.32:0/3096549268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03180ac790 con 0x7f03280725e0 2024-03-16T20:02:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f032d242640 1 -- 172.21.15.32:0/3096549268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180acb20 con 0x7f03280725e0 2024-03-16T20:02:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- 172.21.15.32:0/3096549268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03280725e0 msgr2=0x7f0328073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 --2- 172.21.15.32:0/3096549268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03280725e0 0x7f0328073070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f03280c4a30 tx=0x7f031801c3d0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- 172.21.15.32:0/3096549268 shutdown_connections 2024-03-16T20:02:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 --2- 172.21.15.32:0/3096549268 >> v2:172.21.15.59:3300/0 conn(0x7f032815a420 0x7f032815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 --2- 172.21.15.32:0/3096549268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03280725e0 0x7f0328073070 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- 172.21.15.32:0/3096549268 >> 172.21.15.32:0/3096549268 conn(0x7f032806c0d0 msgr2=0x7f032806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- 172.21.15.32:0/3096549268 shutdown_connections 2024-03-16T20:02:08.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- 172.21.15.32:0/3096549268 wait complete. 2024-03-16T20:02:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 Processor -- start 2024-03-16T20:02:08.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.371+0000 7f03304a8640 1 -- start start 2024-03-16T20:02:08.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280725e0 0x7f0328147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f032815a420 0x7f03281462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032815d0f0 con 0x7f0328150d70 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0328144d80 con 0x7f03280725e0 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f03304a8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0328149ea0 con 0x7f032815a420 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f032da43640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f032da43640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46498/0 (socket says 172.21.15.32:46498) 2024-03-16T20:02:08.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f032da43640 1 -- 172.21.15.32:0/2160056015 learned_addr learned my addr 172.21.15.32:0/2160056015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:08.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f032ea45640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f032815a420 0x7f03281462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.375+0000 7f032e244640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280725e0 0x7f0328147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.379+0000 7f032da43640 1 -- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f032815a420 msgr2=0x7f03281462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.379+0000 7f032da43640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f032815a420 0x7f03281462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.379+0000 7f032da43640 1 -- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280725e0 msgr2=0x7f0328147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.379+0000 7f032da43640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280725e0 0x7f0328147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.379+0000 7f032da43640 1 -- 172.21.15.32:0/2160056015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180094d0 con 0x7f0328150d70 2024-03-16T20:02:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.383+0000 7f032da43640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0318018fe0 tx=0x7f0318019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180b5640 con 0x7f0328150d70 2024-03-16T20:02:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03180b57a0 con 0x7f0328150d70 2024-03-16T20:02:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032815d560 con 0x7f0328150d70 2024-03-16T20:02:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180b5b50 con 0x7f0328150d70 2024-03-16T20:02:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032815cb30 con 0x7f0328150d70 2024-03-16T20:02:08.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.387+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0328068000 con 0x7f0328150d70 2024-03-16T20:02:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.407+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f03180b3070 con 0x7f0328150d70 2024-03-16T20:02:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.407+0000 7f031effd640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 0x7f030c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.407+0000 7f032e244640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 0x7f030c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.407+0000 7f032e244640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 0x7f030c069350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f0324009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.723+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f03180ef920 con 0x7f0328150d70 2024-03-16T20:02:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.723+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f03180ebbc0 con 0x7f0328150d70 2024-03-16T20:02:08.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.971+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03280722b0 con 0x7f0328150d70 2024-03-16T20:02:08.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.987+0000 7f031effd640 1 -- 172.21.15.32:0/2160056015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f03180b1090 con 0x7f0328150d70 2024-03-16T20:02:08.966 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:08.966 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:08.966 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:08.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 msgr2=0x7f030c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 0x7f030c069350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f0324009040 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 msgr2=0x7f0328145540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0318018fe0 tx=0x7f0318019170 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 shutdown_connections 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f030c066e90 0x7f030c069350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f032815a420 0x7f03281462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0328150d70 0x7f0328145540 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 --2- 172.21.15.32:0/2160056015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280725e0 0x7f0328147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 >> 172.21.15.32:0/2160056015 conn(0x7f032806c0d0 msgr2=0x7f032814a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 shutdown_connections 2024-03-16T20:02:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:08.991+0000 7f03304a8640 1 -- 172.21.15.32:0/2160056015 wait complete. 2024-03-16T20:02:09.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:09.593 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:10.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 Processor -- start 2024-03-16T20:02:10.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 -- start start 2024-03-16T20:02:10.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd084072c70 0x7fd0840c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:10.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08406e6a0 con 0x7fd08406bae0 2024-03-16T20:02:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08406c250 con 0x7fd084073050 2024-03-16T20:02:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08c631640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd084156f20 con 0x7fd084072c70 2024-03-16T20:02:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08abce640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd08406bae0 0x7fd0841617c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55204/0 (socket says 172.21.15.32:55204) 2024-03-16T20:02:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08abce640 1 -- 172.21.15.32:0/2289596923 learned_addr learned my addr 172.21.15.32:0/2289596923 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.727+0000 7fd08a3cd640 1 --2- 172.21.15.32:0/2289596923 >> v2:172.21.15.59:3300/0 conn(0x7fd084072c70 0x7fd0840c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:10.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.731+0000 7fd0893cb640 1 -- 172.21.15.32:0/2289596923 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498102528 0 0) 0x7fd08406e6a0 con 0x7fd08406bae0 2024-03-16T20:02:10.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.731+0000 7fd0893cb640 1 -- 172.21.15.32:0/2289596923 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd070003530 con 0x7fd08406bae0 2024-03-16T20:02:10.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.735+0000 7fd08a3cd640 1 -- 172.21.15.32:0/2289596923 >> v1:172.21.15.59:6789/0 conn(0x7fd084073050 legacy=0x7fd084150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:10.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.735+0000 7fd08a3cd640 1 -- 172.21.15.32:0/2289596923 >> v1:172.21.15.32:6789/0 conn(0x7fd08406bae0 legacy=0x7fd0841617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:10.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.735+0000 7fd08a3cd640 1 -- 172.21.15.32:0/2289596923 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08406c510 con 0x7fd084072c70 2024-03-16T20:02:10.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.751+0000 7fd08a3cd640 1 --2- 172.21.15.32:0/2289596923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084072c70 0x7fd0840c0f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd0800097c0 tx=0x7fd08001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec42808253101b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:10.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.759+0000 7fd0893cb640 1 -- 172.21.15.32:0/2289596923 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800aa5f0 con 0x7fd084072c70 2024-03-16T20:02:10.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.759+0000 7fd0893cb640 1 -- 172.21.15.32:0/2289596923 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0800aa750 con 0x7fd084072c70 2024-03-16T20:02:10.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.759+0000 7fd0893cb640 1 -- 172.21.15.32:0/2289596923 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800aaae0 con 0x7fd084072c70 2024-03-16T20:02:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- 172.21.15.32:0/2289596923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084072c70 msgr2=0x7fd0840c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 --2- 172.21.15.32:0/2289596923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084072c70 0x7fd0840c0f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd0800097c0 tx=0x7fd08001c470 comp rx=0 tx=0).stop 2024-03-16T20:02:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- 172.21.15.32:0/2289596923 shutdown_connections 2024-03-16T20:02:10.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 --2- 172.21.15.32:0/2289596923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084072c70 0x7fd0840c0f90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:10.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- 172.21.15.32:0/2289596923 >> 172.21.15.32:0/2289596923 conn(0x7fd08406afc0 msgr2=0x7fd08406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:10.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- 172.21.15.32:0/2289596923 shutdown_connections 2024-03-16T20:02:10.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- 172.21.15.32:0/2289596923 wait complete. 2024-03-16T20:02:10.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 Processor -- start 2024-03-16T20:02:10.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- start start 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 0x7fd08413c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 0x7fd0840caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 0x7fd0840caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08413d870 con 0x7fd084072c70 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084145760 con 0x7fd084073050 2024-03-16T20:02:10.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08c631640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd084147850 con 0x7fd08406bae0 2024-03-16T20:02:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.763+0000 7fd08a3cd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 0x7fd08413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.767+0000 7fd089bcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 0x7fd0840caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.767+0000 7fd08abce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 0x7fd0840caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.767+0000 7fd08a3cd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 0x7fd08413c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:41324/0 (socket says 172.21.15.32:41324) 2024-03-16T20:02:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.767+0000 7fd08abce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 0x7fd0840caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:55512/0 (socket says 172.21.15.32:55512) 2024-03-16T20:02:10.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.767+0000 7fd08a3cd640 1 -- 172.21.15.32:0/3029958612 learned_addr learned my addr 172.21.15.32:0/3029958612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:10.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.771+0000 7fd089bcc640 1 -- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 msgr2=0x7fd08413c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:10.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.771+0000 7fd089bcc640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 0x7fd08413c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:10.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.771+0000 7fd089bcc640 1 -- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 msgr2=0x7fd0840caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:10.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.771+0000 7fd089bcc640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 0x7fd0840caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:10.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.771+0000 7fd089bcc640 1 -- 172.21.15.32:0/3029958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080009420 con 0x7fd084072c70 2024-03-16T20:02:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.775+0000 7fd089bcc640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 0x7fd0840caa40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c390 tx=0x7fd07400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:10.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740052d0 con 0x7fd084072c70 2024-03-16T20:02:10.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074005430 con 0x7fd084072c70 2024-03-16T20:02:10.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074007260 con 0x7fd084072c70 2024-03-16T20:02:10.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0840ca280 con 0x7fd084072c70 2024-03-16T20:02:10.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08413d2b0 con 0x7fd084072c70 2024-03-16T20:02:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:10.779+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0840ca0e0 con 0x7fd084072c70 2024-03-16T20:02:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.071+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd0740073c0 con 0x7fd084072c70 2024-03-16T20:02:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.071+0000 7fd07affd640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 0x7fd068069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.071+0000 7fd08a3cd640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 0x7fd068069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.075+0000 7fd08a3cd640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 0x7fd068069350 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd08001c4b0 tx=0x7fd0800a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.099+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd074054ec0 con 0x7fd084072c70 2024-03-16T20:02:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.099+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd074051160 con 0x7fd084072c70 2024-03-16T20:02:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.355+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0840ca0e0 con 0x7fd084072c70 2024-03-16T20:02:11.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.375+0000 7fd07affd640 1 -- 172.21.15.32:0/3029958612 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fd0840ca0e0 con 0x7fd084072c70 2024-03-16T20:02:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:11.354 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:11.354 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 msgr2=0x7fd068069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 0x7fd068069350 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd08001c4b0 tx=0x7fd0800a7000 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 msgr2=0x7fd0840caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 0x7fd0840caa40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c390 tx=0x7fd07400c850 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 shutdown_connections 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd068066e90 0x7fd068069350 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd084073050 0x7fd0840caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084072c70 0x7fd0840caa40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 --2- 172.21.15.32:0/3029958612 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd08406bae0 0x7fd08413c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 >> 172.21.15.32:0/3029958612 conn(0x7fd08406afc0 msgr2=0x7fd084150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 shutdown_connections 2024-03-16T20:02:11.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:11.379+0000 7fd08c631640 1 -- 172.21.15.32:0/3029958612 wait complete. 2024-03-16T20:02:11.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:11.983 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:12.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:02:12.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:02:12.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 Processor -- start 2024-03-16T20:02:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 -- start start 2024-03-16T20:02:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b84073050 0x7f2b84150c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8406e6a0 con 0x7f2b84072c70 2024-03-16T20:02:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8406c250 con 0x7f2b840c16a0 2024-03-16T20:02:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.111+0000 7f2b891ca640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b84156f20 con 0x7f2b84073050 2024-03-16T20:02:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.115+0000 7f2b8259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b84073050 0x7f2b84150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.115+0000 7f2b8259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b84073050 0x7f2b84150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50166/0 (socket says 172.21.15.32:50166) 2024-03-16T20:02:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.115+0000 7f2b8259c640 1 -- 172.21.15.32:0/1953940899 learned_addr learned my addr 172.21.15.32:0/1953940899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.115+0000 7f2b8359e640 1 --1- 172.21.15.32:0/1953940899 >> v1:172.21.15.59:6789/0 conn(0x7f2b840c16a0 0x7f2b84151160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:60144/0 (socket says 172.21.15.32:60144) 2024-03-16T20:02:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.119+0000 7f2b8259c640 1 -- 172.21.15.32:0/1953940899 >> v1:172.21.15.59:6789/0 conn(0x7f2b840c16a0 legacy=0x7f2b84151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.119+0000 7f2b8259c640 1 -- 172.21.15.32:0/1953940899 >> v1:172.21.15.32:6789/0 conn(0x7f2b84072c70 legacy=0x7f2b840c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.119+0000 7f2b8259c640 1 -- 172.21.15.32:0/1953940899 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8406c510 con 0x7f2b84073050 2024-03-16T20:02:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.123+0000 7f2b8259c640 1 --2- 172.21.15.32:0/1953940899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b84150c20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009870 tx=0x7f2b7801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4741c955c30c42e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b81d9b640 1 -- 172.21.15.32:0/1953940899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780aa650 con 0x7f2b84073050 2024-03-16T20:02:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b81d9b640 1 -- 172.21.15.32:0/1953940899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b780aa7b0 con 0x7f2b84073050 2024-03-16T20:02:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b81d9b640 1 -- 172.21.15.32:0/1953940899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780aab40 con 0x7f2b84073050 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 -- 172.21.15.32:0/1953940899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 msgr2=0x7f2b84150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 --2- 172.21.15.32:0/1953940899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b84150c20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009870 tx=0x7f2b7801c4f0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 -- 172.21.15.32:0/1953940899 shutdown_connections 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 --2- 172.21.15.32:0/1953940899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b84150c20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 -- 172.21.15.32:0/1953940899 >> 172.21.15.32:0/1953940899 conn(0x7f2b8406afc0 msgr2=0x7f2b8406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 -- 172.21.15.32:0/1953940899 shutdown_connections 2024-03-16T20:02:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.127+0000 7f2b891ca640 1 -- 172.21.15.32:0/1953940899 wait complete. 2024-03-16T20:02:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 Processor -- start 2024-03-16T20:02:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 -- start start 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b84072c70 0x7f2b84147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2b840c16a0 0x7f2b840ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c9d70 con 0x7f2b84073050 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84145270 con 0x7f2b84072c70 2024-03-16T20:02:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b891ca640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84145550 con 0x7f2b840c16a0 2024-03-16T20:02:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b8259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b8259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50182/0 (socket says 172.21.15.32:50182) 2024-03-16T20:02:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b8259c640 1 -- 172.21.15.32:0/2062600855 learned_addr learned my addr 172.21.15.32:0/2062600855 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b82d9d640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b84072c70 0x7f2b84147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.131+0000 7f2b8359e640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2b840c16a0 0x7f2b840ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 -- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2b840c16a0 msgr2=0x7f2b840ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2b840c16a0 0x7f2b840ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 -- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b84072c70 msgr2=0x7f2b84147d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b84072c70 0x7f2b84147d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 -- 172.21.15.32:0/2062600855 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780094d0 con 0x7f2b84073050 2024-03-16T20:02:13.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.139+0000 7f2b8259c640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2b780ac040 tx=0x7f2b7801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780b38c0 con 0x7f2b84073050 2024-03-16T20:02:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b780b3a20 con 0x7f2b84073050 2024-03-16T20:02:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b84145830 con 0x7f2b84073050 2024-03-16T20:02:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780b3dd0 con 0x7f2b84073050 2024-03-16T20:02:13.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b84145df0 con 0x7f2b84073050 2024-03-16T20:02:13.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.147+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b840c9ee0 con 0x7f2b84073050 2024-03-16T20:02:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.183+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2b780c6390 con 0x7f2b84073050 2024-03-16T20:02:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.183+0000 7f2b6b7fe640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 0x7f2b70069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.183+0000 7f2b82d9d640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 0x7f2b70069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.183+0000 7f2b82d9d640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 0x7f2b70069380 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2b74007fd0 tx=0x7f2b74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:13.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.471+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2b780f00f0 con 0x7f2b84073050 2024-03-16T20:02:13.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.471+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2b780ec390 con 0x7f2b84073050 2024-03-16T20:02:13.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.739+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b840674e0 con 0x7f2b84073050 2024-03-16T20:02:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.799+0000 7f2b6b7fe640 1 -- 172.21.15.32:0/2062600855 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f2b780efa50 con 0x7f2b84073050 2024-03-16T20:02:13.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:13.779 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:13.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 msgr2=0x7f2b70069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 0x7f2b70069380 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2b74007fd0 tx=0x7f2b74009040 comp rx=0 tx=0).stop 2024-03-16T20:02:13.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 msgr2=0x7f2b840c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:13.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2b780ac040 tx=0x7f2b7801ad30 comp rx=0 tx=0).stop 2024-03-16T20:02:13.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 shutdown_connections 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2b70066ec0 0x7f2b70069380 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2b840c16a0 0x7f2b840ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b84073050 0x7f2b840c5150 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 --2- 172.21.15.32:0/2062600855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b84072c70 0x7f2b84147d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 >> 172.21.15.32:0/2062600855 conn(0x7f2b8406afc0 msgr2=0x7f2b8406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 shutdown_connections 2024-03-16T20:02:13.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:13.803+0000 7f2b891ca640 1 -- 172.21.15.32:0/2062600855 wait complete. 2024-03-16T20:02:14.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:14.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:15.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:15.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 Processor -- start 2024-03-16T20:02:15.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 -- start start 2024-03-16T20:02:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f1100073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f110015a420 0x7f110015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1100151020 con 0x7f11000725e0 2024-03-16T20:02:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110015a000 con 0x7f110015a420 2024-03-16T20:02:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1106eb4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110015adf0 con 0x7f1100150c40 2024-03-16T20:02:15.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.535+0000 7f1105451640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f110015a420 0x7f110015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.539+0000 7f1104c50640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f1100073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.539+0000 7f10fffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f11000725e0 0x7f1100070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:60146/0 (socket says 172.21.15.32:60146) 2024-03-16T20:02:15.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.539+0000 7f10fffff640 1 -- 172.21.15.32:0/2553079842 learned_addr learned my addr 172.21.15.32:0/2553079842 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.543+0000 7f1105451640 1 -- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 msgr2=0x7f1100073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.543+0000 7f1105451640 1 --2- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f1100073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.543+0000 7f1105451640 1 -- 172.21.15.32:0/2553079842 >> v1:172.21.15.59:6789/0 conn(0x7f11000725e0 legacy=0x7f1100070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.543+0000 7f1105451640 1 -- 172.21.15.32:0/2553079842 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100072da0 con 0x7f110015a420 2024-03-16T20:02:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.547+0000 7f1105451640 1 --2- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110015a8b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3645953b4bb9a14 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.551+0000 7f10ff7fe640 1 -- 172.21.15.32:0/2553079842 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80a5a30 con 0x7f110015a420 2024-03-16T20:02:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.555+0000 7f10ff7fe640 1 -- 172.21.15.32:0/2553079842 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f80a5b90 con 0x7f110015a420 2024-03-16T20:02:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.555+0000 7f10ff7fe640 1 -- 172.21.15.32:0/2553079842 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80ad440 con 0x7f110015a420 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 msgr2=0x7f110015a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110015a8b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f801c970 comp rx=0 tx=0).stop 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- 172.21.15.32:0/2553079842 shutdown_connections 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110015a8b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2553079842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f1100073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- 172.21.15.32:0/2553079842 >> 172.21.15.32:0/2553079842 conn(0x7f110006c0d0 msgr2=0x7f110006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- 172.21.15.32:0/2553079842 shutdown_connections 2024-03-16T20:02:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- 172.21.15.32:0/2553079842 wait complete. 2024-03-16T20:02:15.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 Processor -- start 2024-03-16T20:02:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- start start 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 0x7f1100153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f110014e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1100151410 con 0x7f11000725e0 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1100148d70 con 0x7f110015a420 2024-03-16T20:02:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.559+0000 7f1106eb4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11001f6960 con 0x7f1100150c40 2024-03-16T20:02:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.563+0000 7f1105451640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.563+0000 7f1104c50640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 0x7f1100153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.563+0000 7f1105451640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:55542/0 (socket says 172.21.15.32:55542) 2024-03-16T20:02:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.563+0000 7f1105451640 1 -- 172.21.15.32:0/2871697951 learned_addr learned my addr 172.21.15.32:0/2871697951 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.563+0000 7f10fffff640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f110014e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1105451640 1 -- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 msgr2=0x7f110014e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1105451640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f110014e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1105451640 1 -- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 msgr2=0x7f1100153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1105451640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 0x7f1100153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1105451640 1 -- 172.21.15.32:0/2871697951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f8009a70 con 0x7f110015a420 2024-03-16T20:02:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.567+0000 7f1104c50640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 0x7f1100153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:02:15.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.571+0000 7f1105451640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f8019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.579+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80a5a30 con 0x7f110015a420 2024-03-16T20:02:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.579+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1100149010 con 0x7f110015a420 2024-03-16T20:02:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.579+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11001f6c40 con 0x7f110015a420 2024-03-16T20:02:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.579+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1100149250 con 0x7f110015a420 2024-03-16T20:02:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.807+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f8009450 con 0x7f110015a420 2024-03-16T20:02:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.807+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80b9650 con 0x7f110015a420 2024-03-16T20:02:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.915+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f10f80b9870 con 0x7f110015a420 2024-03-16T20:02:15.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.915+0000 7f10fd7fa640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 0x7f10e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.915+0000 7f1104c50640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 0x7f10e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.915+0000 7f1104c50640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 0x7f10e4069300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009f50 tx=0x7f10f4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:15.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.927+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f10f80f11a0 con 0x7f110015a420 2024-03-16T20:02:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:15.955+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f10f80ed440 con 0x7f110015a420 2024-03-16T20:02:16.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.227+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1100151840 con 0x7f110015a420 2024-03-16T20:02:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.263+0000 7f10fd7fa640 1 -- 172.21.15.32:0/2871697951 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1100151840 con 0x7f110015a420 2024-03-16T20:02:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:16.241 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 msgr2=0x7f10e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 0x7f10e4069300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009f50 tx=0x7f10f4009190 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 msgr2=0x7f110014e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f8019710 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 shutdown_connections 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10e4066e40 0x7f10e4069300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f110015a420 0x7f110014e910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1100150c40 0x7f110014e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 --2- 172.21.15.32:0/2871697951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11000725e0 0x7f1100153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 >> 172.21.15.32:0/2871697951 conn(0x7f110006c0d0 msgr2=0x7f110006f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 shutdown_connections 2024-03-16T20:02:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:16.267+0000 7f1106eb4640 1 -- 172.21.15.32:0/2871697951 wait complete. 2024-03-16T20:02:16.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:16.869 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:17.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:17.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:17.999+0000 7f8dde830640 1 Processor -- start 2024-03-16T20:02:17.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 -- start start 2024-03-16T20:02:17.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8dd80725e0 0x7f8dd8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:17.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd815a420 0x7f8dd815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:17.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8072dd0 con 0x7f8dd8150d70 2024-03-16T20:02:17.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd815d790 con 0x7f8dd815a420 2024-03-16T20:02:17.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dde830640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd815da50 con 0x7f8dd80725e0 2024-03-16T20:02:17.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8ddcdcd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd815a420 0x7f8dd815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:17.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8ddcdcd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd815a420 0x7f8dd815a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50202/0 (socket says 172.21.15.32:50202) 2024-03-16T20:02:17.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/2031739670 learned_addr learned my addr 172.21.15.32:0/2031739670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:17.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.003+0000 7f8dd77fe640 1 --2- 172.21.15.32:0/2031739670 >> v2:172.21.15.59:3300/0 conn(0x7f8dd80725e0 0x7f8dd8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.007+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/2031739670 >> v2:172.21.15.59:3300/0 conn(0x7f8dd80725e0 msgr2=0x7f8dd8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.007+0000 7f8ddcdcd640 1 --2- 172.21.15.32:0/2031739670 >> v2:172.21.15.59:3300/0 conn(0x7f8dd80725e0 0x7f8dd8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.007+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/2031739670 >> v1:172.21.15.59:6789/0 conn(0x7f8dd8150d70 legacy=0x7f8dd8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:17.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.007+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/2031739670 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd815dd30 con 0x7f8dd815a420 2024-03-16T20:02:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.011+0000 7f8ddcdcd640 1 --2- 172.21.15.32:0/2031739670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd815a830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8dd806ef10 tx=0x7f8dd001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff853bc78a1e780d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dd6ffd640 1 -- 172.21.15.32:0/2031739670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00a5720 con 0x7f8dd815a420 2024-03-16T20:02:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dd6ffd640 1 -- 172.21.15.32:0/2031739670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd00a5880 con 0x7f8dd815a420 2024-03-16T20:02:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dd6ffd640 1 -- 172.21.15.32:0/2031739670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00a5c10 con 0x7f8dd815a420 2024-03-16T20:02:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 -- 172.21.15.32:0/2031739670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 msgr2=0x7f8dd815a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 --2- 172.21.15.32:0/2031739670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd815a830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8dd806ef10 tx=0x7f8dd001c860 comp rx=0 tx=0).stop 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 -- 172.21.15.32:0/2031739670 shutdown_connections 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 --2- 172.21.15.32:0/2031739670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd815a830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 --2- 172.21.15.32:0/2031739670 >> v2:172.21.15.59:3300/0 conn(0x7f8dd80725e0 0x7f8dd8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 -- 172.21.15.32:0/2031739670 >> 172.21.15.32:0/2031739670 conn(0x7f8dd806c0d0 msgr2=0x7f8dd806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 -- 172.21.15.32:0/2031739670 shutdown_connections 2024-03-16T20:02:17.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.015+0000 7f8dde830640 1 -- 172.21.15.32:0/2031739670 wait complete. 2024-03-16T20:02:17.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 Processor -- start 2024-03-16T20:02:17.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 -- start start 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dd80725e0 0x7f8dd815c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8dd8150d70 0x7f8dd814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd81f6cc0 con 0x7f8dd815a420 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd81f6f80 con 0x7f8dd80725e0 2024-03-16T20:02:17.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dde830640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd814b0c0 con 0x7f8dd8150d70 2024-03-16T20:02:17.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8ddcdcd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:17.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8ddcdcd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50206/0 (socket says 172.21.15.32:50206) 2024-03-16T20:02:17.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/405345859 learned_addr learned my addr 172.21.15.32:0/405345859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:17.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.019+0000 7f8dd7fff640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dd80725e0 0x7f8dd815c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:18.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8dd77fe640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8dd8150d70 0x7f8dd814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:18.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8dd8150d70 msgr2=0x7f8dd814c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:18.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8ddcdcd640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8dd8150d70 0x7f8dd814c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dd80725e0 msgr2=0x7f8dd815c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:18.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8ddcdcd640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dd80725e0 0x7f8dd815c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.023+0000 7f8ddcdcd640 1 -- 172.21.15.32:0/405345859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0009b30 con 0x7f8dd815a420 2024-03-16T20:02:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.027+0000 7f8ddcdcd640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8dd001af10 tx=0x7f8dd0019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.031+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00b8670 con 0x7f8dd815a420 2024-03-16T20:02:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.031+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd00b87d0 con 0x7f8dd815a420 2024-03-16T20:02:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.031+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd815d560 con 0x7f8dd815a420 2024-03-16T20:02:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.031+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00b8b60 con 0x7f8dd815a420 2024-03-16T20:02:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.031+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd814b680 con 0x7f8dd815a420 2024-03-16T20:02:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.035+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd8068000 con 0x7f8dd815a420 2024-03-16T20:02:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.059+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8dd00b8cc0 con 0x7f8dd815a420 2024-03-16T20:02:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.059+0000 7f8dd4ff9640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 0x7f8dc4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.059+0000 7f8dd7fff640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 0x7f8dc4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.059+0000 7f8dd7fff640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 0x7f8dc4069350 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80098e0 tx=0x7f8dc8009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.063+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8dd00f0f00 con 0x7f8dd815a420 2024-03-16T20:02:18.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.391+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8dd00ed1a0 con 0x7f8dd815a420 2024-03-16T20:02:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.627+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dd80c8a70 con 0x7f8dd815a420 2024-03-16T20:02:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.643+0000 7f8dd4ff9640 1 -- 172.21.15.32:0/405345859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8dd00bc090 con 0x7f8dd815a420 2024-03-16T20:02:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:18.623 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 msgr2=0x7f8dc4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 0x7f8dc4069350 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80098e0 tx=0x7f8dc8009060 comp rx=0 tx=0).stop 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 msgr2=0x7f8dd814cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8dd001af10 tx=0x7f8dd0019600 comp rx=0 tx=0).stop 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 shutdown_connections 2024-03-16T20:02:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8dc4066e90 0x7f8dc4069350 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd815a420 0x7f8dd814cb20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8dd8150d70 0x7f8dd814c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 --2- 172.21.15.32:0/405345859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dd80725e0 0x7f8dd815c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 >> 172.21.15.32:0/405345859 conn(0x7f8dd806c0d0 msgr2=0x7f8dd8147570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 shutdown_connections 2024-03-16T20:02:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:18.647+0000 7f8dde830640 1 -- 172.21.15.32:0/405345859 wait complete. 2024-03-16T20:02:19.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:19.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:20.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 Processor -- start 2024-03-16T20:02:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 -- start start 2024-03-16T20:02:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad8151020 con 0x7fcad8150c40 2024-03-16T20:02:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad8072da0 con 0x7fcad80725e0 2024-03-16T20:02:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadeab0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815a000 con 0x7fcad815ab30 2024-03-16T20:02:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadc84c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcad8150c40 0x7fcad8070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54932/0 (socket says 172.21.15.32:54932) 2024-03-16T20:02:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadc84c640 1 -- 172.21.15.32:0/2861302709 learned_addr learned my addr 172.21.15.32:0/2861302709 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.383+0000 7fcadd04d640 1 --2- 172.21.15.32:0/2861302709 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:20.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.387+0000 7fcad77fe640 1 -- 172.21.15.32:0/2861302709 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574572387 0 0) 0x7fcad8151020 con 0x7fcad8150c40 2024-03-16T20:02:20.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.387+0000 7fcad77fe640 1 -- 172.21.15.32:0/2861302709 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8003530 con 0x7fcad8150c40 2024-03-16T20:02:20.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.391+0000 7fcadd04d640 1 -- 172.21.15.32:0/2861302709 >> v1:172.21.15.59:6789/0 conn(0x7fcad80725e0 legacy=0x7fcad815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.391+0000 7fcadd04d640 1 -- 172.21.15.32:0/2861302709 >> v1:172.21.15.32:6789/0 conn(0x7fcad8150c40 legacy=0x7fcad8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.391+0000 7fcadd04d640 1 -- 172.21.15.32:0/2861302709 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad815e310 con 0x7fcad815ab30 2024-03-16T20:02:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.399+0000 7fcadd04d640 1 --2- 172.21.15.32:0/2861302709 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad8073070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009dc0 tx=0x7fcad001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b54fbb7d360031f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.407+0000 7fcad77fe640 1 -- 172.21.15.32:0/2861302709 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad00a5a50 con 0x7fcad815ab30 2024-03-16T20:02:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.407+0000 7fcad77fe640 1 -- 172.21.15.32:0/2861302709 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad00a5bb0 con 0x7fcad815ab30 2024-03-16T20:02:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.407+0000 7fcad77fe640 1 -- 172.21.15.32:0/2861302709 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad00ad440 con 0x7fcad815ab30 2024-03-16T20:02:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- 172.21.15.32:0/2861302709 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 msgr2=0x7fcad8073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2861302709 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad8073070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009dc0 tx=0x7fcad001c990 comp rx=0 tx=0).stop 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- 172.21.15.32:0/2861302709 shutdown_connections 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2861302709 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad8073070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- 172.21.15.32:0/2861302709 >> 172.21.15.32:0/2861302709 conn(0x7fcad806c0d0 msgr2=0x7fcad806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- 172.21.15.32:0/2861302709 shutdown_connections 2024-03-16T20:02:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- 172.21.15.32:0/2861302709 wait complete. 2024-03-16T20:02:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 Processor -- start 2024-03-16T20:02:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- start start 2024-03-16T20:02:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcad80725e0 0x7fcad8153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad81519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8151550 con 0x7fcad8150c40 2024-03-16T20:02:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad814dfa0 con 0x7fcad80725e0 2024-03-16T20:02:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcadeab0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8149360 con 0x7fcad815ab30 2024-03-16T20:02:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.411+0000 7fcad7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.415+0000 7fcad7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49276/0 (socket says 172.21.15.32:49276) 2024-03-16T20:02:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.415+0000 7fcad7fff640 1 -- 172.21.15.32:0/2763003620 learned_addr learned my addr 172.21.15.32:0/2763003620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.415+0000 7fcadc84c640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcad80725e0 0x7fcad8153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.415+0000 7fcadd04d640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad81519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.419+0000 7fcad7fff640 1 -- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 msgr2=0x7fcad81519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.419+0000 7fcad7fff640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad81519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.419+0000 7fcad7fff640 1 -- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcad80725e0 msgr2=0x7fcad8153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.419+0000 7fcad7fff640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcad80725e0 0x7fcad8153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.419+0000 7fcad7fff640 1 -- 172.21.15.32:0/2763003620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0009a70 con 0x7fcad8150c40 2024-03-16T20:02:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.423+0000 7fcad7fff640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c420 tx=0x7fcac400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:20.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac40052d0 con 0x7fcad8150c40 2024-03-16T20:02:20.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4005430 con 0x7fcad8150c40 2024-03-16T20:02:20.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad815d560 con 0x7fcad8150c40 2024-03-16T20:02:20.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4007260 con 0x7fcad8150c40 2024-03-16T20:02:20.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad8149920 con 0x7fcad8150c40 2024-03-16T20:02:20.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.427+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8151700 con 0x7fcad8150c40 2024-03-16T20:02:20.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.455+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcac40073c0 con 0x7fcad8150c40 2024-03-16T20:02:20.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.455+0000 7fcad57fa640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 0x7fcabc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:20.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.455+0000 7fcadc84c640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 0x7fcabc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:20.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.455+0000 7fcadc84c640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 0x7fcabc069350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc000bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:20.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.455+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fcac4007670 con 0x7fcad8150c40 2024-03-16T20:02:20.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.723+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fcac4051310 con 0x7fcad8150c40 2024-03-16T20:02:20.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.935+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcad8068000 con 0x7fcad8150c40 2024-03-16T20:02:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.959+0000 7fcad57fa640 1 -- 172.21.15.32:0/2763003620 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fcac4020020 con 0x7fcad8150c40 2024-03-16T20:02:20.937 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 msgr2=0x7fcabc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 0x7fcabc069350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc000bd0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 msgr2=0x7fcad80c5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c420 tx=0x7fcac400c8e0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 shutdown_connections 2024-03-16T20:02:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fcabc066e90 0x7fcabc069350 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcad815ab30 0x7fcad81519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcad8150c40 0x7fcad80c5d50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 --2- 172.21.15.32:0/2763003620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcad80725e0 0x7fcad8153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 >> 172.21.15.32:0/2763003620 conn(0x7fcad806c0d0 msgr2=0x7fcad814a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 shutdown_connections 2024-03-16T20:02:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:20.963+0000 7fcadeab0640 1 -- 172.21.15.32:0/2763003620 wait complete. 2024-03-16T20:02:21.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:21.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:22.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:22.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.691+0000 7fdf98231640 1 Processor -- start 2024-03-16T20:02:22.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.691+0000 7fdf98231640 1 -- start start 2024-03-16T20:02:22.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf98231640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf90150c40 0x7fdf90073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf98231640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf9015a420 0x7fdf9015a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf98231640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf90151020 con 0x7fdf900725e0 2024-03-16T20:02:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf98231640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9015a000 con 0x7fdf90150c40 2024-03-16T20:02:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf98231640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9015add0 con 0x7fdf9015a420 2024-03-16T20:02:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf967ce640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf9015a420 0x7fdf9015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:22.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf967ce640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf9015a420 0x7fdf9015a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:48470/0 (socket says 172.21.15.32:48470) 2024-03-16T20:02:22.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.695+0000 7fdf967ce640 1 -- 172.21.15.32:0/355446244 learned_addr learned my addr 172.21.15.32:0/355446244 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.703+0000 7fdf967ce640 1 -- 172.21.15.32:0/355446244 >> v2:172.21.15.32:3300/0 conn(0x7fdf90150c40 msgr2=0x7fdf90073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.703+0000 7fdf967ce640 1 --2- 172.21.15.32:0/355446244 >> v2:172.21.15.32:3300/0 conn(0x7fdf90150c40 0x7fdf90073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.703+0000 7fdf967ce640 1 -- 172.21.15.32:0/355446244 >> v1:172.21.15.32:6789/0 conn(0x7fdf900725e0 legacy=0x7fdf90070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.703+0000 7fdf967ce640 1 -- 172.21.15.32:0/355446244 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf90072da0 con 0x7fdf9015a420 2024-03-16T20:02:22.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.707+0000 7fdf967ce640 1 --2- 172.21.15.32:0/355446244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf9015a890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf900ca290 tx=0x7fdf8801c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ef94a91e57b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf94fcb640 1 -- 172.21.15.32:0/355446244 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880a5870 con 0x7fdf9015a420 2024-03-16T20:02:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf94fcb640 1 -- 172.21.15.32:0/355446244 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf880a59d0 con 0x7fdf9015a420 2024-03-16T20:02:22.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf94fcb640 1 -- 172.21.15.32:0/355446244 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880a5d60 con 0x7fdf9015a420 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 -- 172.21.15.32:0/355446244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 msgr2=0x7fdf9015a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 --2- 172.21.15.32:0/355446244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf9015a890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf900ca290 tx=0x7fdf8801c9b0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 -- 172.21.15.32:0/355446244 shutdown_connections 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 --2- 172.21.15.32:0/355446244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf9015a890 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 --2- 172.21.15.32:0/355446244 >> v2:172.21.15.32:3300/0 conn(0x7fdf90150c40 0x7fdf90073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 -- 172.21.15.32:0/355446244 >> 172.21.15.32:0/355446244 conn(0x7fdf9006c0d0 msgr2=0x7fdf9006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 -- 172.21.15.32:0/355446244 shutdown_connections 2024-03-16T20:02:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.715+0000 7fdf98231640 1 -- 172.21.15.32:0/355446244 wait complete. 2024-03-16T20:02:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 Processor -- start 2024-03-16T20:02:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 -- start start 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf90150c40 0x7fdf901469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf90143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf901518b0 con 0x7fdf900725e0 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9014a0d0 con 0x7fdf9015a420 2024-03-16T20:02:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf98231640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf901f3060 con 0x7fdf90150c40 2024-03-16T20:02:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf95fcd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf95fcd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49284/0 (socket says 172.21.15.32:49284) 2024-03-16T20:02:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.719+0000 7fdf95fcd640 1 -- 172.21.15.32:0/2452769683 learned_addr learned my addr 172.21.15.32:0/2452769683 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf967ce640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf90143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf957cc640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf90150c40 0x7fdf901469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:22.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf95fcd640 1 -- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf90150c40 msgr2=0x7fdf901469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:22.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf95fcd640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf90150c40 0x7fdf901469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf95fcd640 1 -- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 msgr2=0x7fdf90143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf95fcd640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf90143f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.723+0000 7fdf95fcd640 1 -- 172.21.15.32:0/2452769683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88009b30 con 0x7fdf900725e0 2024-03-16T20:02:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.727+0000 7fdf95fcd640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00b410 tx=0x7fdf8c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c011830 con 0x7fdf900725e0 2024-03-16T20:02:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c011990 con 0x7fdf900725e0 2024-03-16T20:02:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c011d40 con 0x7fdf900725e0 2024-03-16T20:02:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf90149df0 con 0x7fdf900725e0 2024-03-16T20:02:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf901464a0 con 0x7fdf900725e0 2024-03-16T20:02:22.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.735+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf90068000 con 0x7fdf900725e0 2024-03-16T20:02:22.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.775+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdf8c01a400 con 0x7fdf900725e0 2024-03-16T20:02:22.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.775+0000 7fdf867fc640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 0x7fdf74069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:22.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.775+0000 7fdf957cc640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 0x7fdf74069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:22.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:22.775+0000 7fdf957cc640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 0x7fdf74069380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf900c9b70 tx=0x7fdf80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:23.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.067+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdf8c056440 con 0x7fdf900725e0 2024-03-16T20:02:23.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.067+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdf8c051fe0 con 0x7fdf900725e0 2024-03-16T20:02:23.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.371+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf900722b0 con 0x7fdf900725e0 2024-03-16T20:02:23.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.395+0000 7fdf867fc640 1 -- 172.21.15.32:0/2452769683 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fdf8c015090 con 0x7fdf900725e0 2024-03-16T20:02:23.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:23.376 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:23.376 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 msgr2=0x7fdf74069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 0x7fdf74069380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf900c9b70 tx=0x7fdf80008040 comp rx=0 tx=0).stop 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 msgr2=0x7fdf901f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00b410 tx=0x7fdf8c00b8d0 comp rx=0 tx=0).stop 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 shutdown_connections 2024-03-16T20:02:23.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf74066ec0 0x7fdf74069380 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf9015a420 0x7fdf90143f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf90150c40 0x7fdf901469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 --2- 172.21.15.32:0/2452769683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf900725e0 0x7fdf901f26b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 >> 172.21.15.32:0/2452769683 conn(0x7fdf9006c0d0 msgr2=0x7fdf900700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 shutdown_connections 2024-03-16T20:02:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:23.399+0000 7fdf98231640 1 -- 172.21.15.32:0/2452769683 wait complete. 2024-03-16T20:02:24.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:24.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:25.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 Processor -- start 2024-03-16T20:02:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 -- start start 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc1542c0 0x7fdfdc073010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdfdc070360 0x7fdfdc072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc1546a0 con 0x7fdfdc072a70 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc072e80 con 0x7fdfdc070360 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe4238640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc073550 con 0x7fdfdc1542c0 2024-03-16T20:02:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe17d3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdfdc070360 0x7fdfdc072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe17d3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdfdc070360 0x7fdfdc072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49296/0 (socket says 172.21.15.32:49296) 2024-03-16T20:02:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe17d3640 1 -- 172.21.15.32:0/2546318489 learned_addr learned my addr 172.21.15.32:0/2546318489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.139+0000 7fdfe1fd4640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc1542c0 0x7fdfdc073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.143+0000 7fdfe17d3640 1 -- 172.21.15.32:0/2546318489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc1542c0 msgr2=0x7fdfdc073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.143+0000 7fdfe17d3640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc1542c0 0x7fdfdc073010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.143+0000 7fdfe17d3640 1 -- 172.21.15.32:0/2546318489 >> v1:172.21.15.59:6789/0 conn(0x7fdfdc072a70 legacy=0x7fdfdc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.143+0000 7fdfe17d3640 1 -- 172.21.15.32:0/2546318489 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc159f20 con 0x7fdfdc070360 2024-03-16T20:02:25.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.151+0000 7fdfe17d3640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc072530 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009870 tx=0x7fdfcc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d84714dce557aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:25.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe0fd2640 1 -- 172.21.15.32:0/2546318489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0a8a50 con 0x7fdfdc070360 2024-03-16T20:02:25.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe0fd2640 1 -- 172.21.15.32:0/2546318489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc0a8bb0 con 0x7fdfdc070360 2024-03-16T20:02:25.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe0fd2640 1 -- 172.21.15.32:0/2546318489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0b0440 con 0x7fdfdc070360 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 -- 172.21.15.32:0/2546318489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 msgr2=0x7fdfdc072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc072530 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009870 tx=0x7fdfcc01ed50 comp rx=0 tx=0).stop 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 -- 172.21.15.32:0/2546318489 shutdown_connections 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc072530 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 --2- 172.21.15.32:0/2546318489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc1542c0 0x7fdfdc073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 -- 172.21.15.32:0/2546318489 >> 172.21.15.32:0/2546318489 conn(0x7fdfdc06c0d0 msgr2=0x7fdfdc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 -- 172.21.15.32:0/2546318489 shutdown_connections 2024-03-16T20:02:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.155+0000 7fdfe4238640 1 -- 172.21.15.32:0/2546318489 wait complete. 2024-03-16T20:02:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 Processor -- start 2024-03-16T20:02:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 -- start start 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc072a70 0x7fdfdc146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc1542c0 0x7fdfdc0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc14b110 con 0x7fdfdc070360 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1442c0 con 0x7fdfdc1542c0 2024-03-16T20:02:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe4238640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1445a0 con 0x7fdfdc072a70 2024-03-16T20:02:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe1fd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe1fd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49308/0 (socket says 172.21.15.32:49308) 2024-03-16T20:02:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe1fd4640 1 -- 172.21.15.32:0/4204634131 learned_addr learned my addr 172.21.15.32:0/4204634131 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe17d3640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc072a70 0x7fdfdc146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.159+0000 7fdfe27d5640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc1542c0 0x7fdfdc0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 -- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc072a70 msgr2=0x7fdfdc146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc072a70 0x7fdfdc146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 -- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc1542c0 msgr2=0x7fdfdc0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc1542c0 0x7fdfdc0ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 -- 172.21.15.32:0/4204634131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0094d0 con 0x7fdfdc070360 2024-03-16T20:02:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.167+0000 7fdfe1fd4640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800e870 tx=0x7fdfd800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8009510 con 0x7fdfdc070360 2024-03-16T20:02:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd800eec0 con 0x7fdfdc070360 2024-03-16T20:02:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd800cb40 con 0x7fdfdc070360 2024-03-16T20:02:25.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc144880 con 0x7fdfdc070360 2024-03-16T20:02:25.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc144e40 con 0x7fdfdc070360 2024-03-16T20:02:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.175+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc068000 con 0x7fdfdc070360 2024-03-16T20:02:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.199+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdfd800cca0 con 0x7fdfdc070360 2024-03-16T20:02:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.199+0000 7fdfd27fc640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 0x7fdfc0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.199+0000 7fdfe17d3640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 0x7fdfc0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.203+0000 7fdfe17d3640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 0x7fdfc0069350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0b2000 tx=0x7fdfcc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:25.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.483+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdfd8055e80 con 0x7fdfdc070360 2024-03-16T20:02:25.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.483+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdfd8052120 con 0x7fdfdc070360 2024-03-16T20:02:25.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.831+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfdc15d1d0 con 0x7fdfdc070360 2024-03-16T20:02:25.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.855+0000 7fdfd27fc640 1 -- 172.21.15.32:0/4204634131 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fdfd8017090 con 0x7fdfdc070360 2024-03-16T20:02:25.835 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:25.835 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 msgr2=0x7fdfc0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 0x7fdfc0069350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0b2000 tx=0x7fdfcc0aa070 comp rx=0 tx=0).stop 2024-03-16T20:02:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 msgr2=0x7fdfdc0c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800e870 tx=0x7fdfd800ec70 comp rx=0 tx=0).stop 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 shutdown_connections 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdfc0066e90 0x7fdfc0069350 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc1542c0 0x7fdfdc0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdfdc072a70 0x7fdfdc146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 --2- 172.21.15.32:0/4204634131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc070360 0x7fdfdc0c5c30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 >> 172.21.15.32:0/4204634131 conn(0x7fdfdc06c0d0 msgr2=0x7fdfdc15d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 shutdown_connections 2024-03-16T20:02:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:25.859+0000 7fdfe4238640 1 -- 172.21.15.32:0/4204634131 wait complete. 2024-03-16T20:02:26.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:26.458 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:27.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 Processor -- start 2024-03-16T20:02:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 -- start start 2024-03-16T20:02:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5574158390 0x7f55740c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5574070760 con 0x7f55740c6420 2024-03-16T20:02:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55740731c0 con 0x7f55740c6040 2024-03-16T20:02:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557bb7f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55740734a0 con 0x7f5574158390 2024-03-16T20:02:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557a11c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5574158390 0x7f55740c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557991b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f55740c6040 0x7f5574157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:50024/0 (socket says 172.21.15.32:50024) 2024-03-16T20:02:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.591+0000 7f557991b640 1 -- 172.21.15.32:0/1451541984 learned_addr learned my addr 172.21.15.32:0/1451541984 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.595+0000 7f557a11c640 1 -- 172.21.15.32:0/1451541984 >> v1:172.21.15.59:6789/0 conn(0x7f55740c6040 legacy=0x7f5574157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.595+0000 7f557a11c640 1 -- 172.21.15.32:0/1451541984 >> v1:172.21.15.32:6789/0 conn(0x7f55740c6420 legacy=0x7f55741516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.595+0000 7f557a11c640 1 -- 172.21.15.32:0/1451541984 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557406fe30 con 0x7f5574158390 2024-03-16T20:02:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.599+0000 7f557a11c640 1 --2- 172.21.15.32:0/1451541984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5574158390 0x7f55740c8800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f556c009d90 tx=0x7f556c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d21351bc9e99bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.603+0000 7f5578919640 1 -- 172.21.15.32:0/1451541984 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0a59c0 con 0x7f5574158390 2024-03-16T20:02:27.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f5578919640 1 -- 172.21.15.32:0/1451541984 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c0a5b20 con 0x7f5574158390 2024-03-16T20:02:27.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f5578919640 1 -- 172.21.15.32:0/1451541984 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0ad440 con 0x7f5574158390 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- 172.21.15.32:0/1451541984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5574158390 msgr2=0x7f55740c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 --2- 172.21.15.32:0/1451541984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5574158390 0x7f55740c8800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f556c009d90 tx=0x7f556c01c900 comp rx=0 tx=0).stop 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- 172.21.15.32:0/1451541984 shutdown_connections 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 --2- 172.21.15.32:0/1451541984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5574158390 0x7f55740c8800 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- 172.21.15.32:0/1451541984 >> 172.21.15.32:0/1451541984 conn(0x7f557406c0d0 msgr2=0x7f557406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- 172.21.15.32:0/1451541984 shutdown_connections 2024-03-16T20:02:27.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- 172.21.15.32:0/1451541984 wait complete. 2024-03-16T20:02:27.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 Processor -- start 2024-03-16T20:02:27.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- start start 2024-03-16T20:02:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55740c6040 0x7f557414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5574158390 0x7f55741fa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557415b110 con 0x7f55740c6420 2024-03-16T20:02:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55741f1960 con 0x7f5574158390 2024-03-16T20:02:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.607+0000 7f557bb7f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557415b690 con 0x7f55740c6040 2024-03-16T20:02:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.611+0000 7f557911a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.611+0000 7f557911a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49324/0 (socket says 172.21.15.32:49324) 2024-03-16T20:02:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.611+0000 7f557911a640 1 -- 172.21.15.32:0/2075125980 learned_addr learned my addr 172.21.15.32:0/2075125980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:27.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.611+0000 7f557a11c640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5574158390 0x7f55741fa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:27.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.611+0000 7f557991b640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55740c6040 0x7f557414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:27.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.615+0000 7f557911a640 1 -- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55740c6040 msgr2=0x7f557414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.615+0000 7f557911a640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55740c6040 0x7f557414e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.615+0000 7f557911a640 1 -- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5574158390 msgr2=0x7f55741fa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.615+0000 7f557911a640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5574158390 0x7f55741fa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.615+0000 7f557911a640 1 -- 172.21.15.32:0/2075125980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c009a70 con 0x7f55740c6420 2024-03-16T20:02:27.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.619+0000 7f557911a640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f556400c440 tx=0x7f556400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:27.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.623+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564005480 con 0x7f55740c6420 2024-03-16T20:02:27.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.623+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5564003600 con 0x7f55740c6420 2024-03-16T20:02:27.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.623+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55741fb6a0 con 0x7f55740c6420 2024-03-16T20:02:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.623+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564003990 con 0x7f55740c6420 2024-03-16T20:02:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.623+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5574159e00 con 0x7f55740c6420 2024-03-16T20:02:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.627+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5574068000 con 0x7f55740c6420 2024-03-16T20:02:27.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.663+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5564012400 con 0x7f55740c6420 2024-03-16T20:02:27.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.663+0000 7f556a7fc640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 0x7f5558069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:27.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.663+0000 7f557991b640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 0x7f5558069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:27.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.663+0000 7f557991b640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 0x7f5558069330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5570000f70 tx=0x7f5570007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.667+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f55640010d0 con 0x7f55740c6420 2024-03-16T20:02:27.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:27.979+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5564051590 con 0x7f55740c6420 2024-03-16T20:02:28.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.203+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557415b7d0 con 0x7f55740c6420 2024-03-16T20:02:28.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.251+0000 7f556a7fc640 1 -- 172.21.15.32:0/2075125980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f5564021070 con 0x7f55740c6420 2024-03-16T20:02:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:28.231 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:28.231 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 msgr2=0x7f5558069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 0x7f5558069330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5570000f70 tx=0x7f5570007e90 comp rx=0 tx=0).stop 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 msgr2=0x7f55741f86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f556400c440 tx=0x7f556400c900 comp rx=0 tx=0).stop 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 shutdown_connections 2024-03-16T20:02:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5558066e70 0x7f5558069330 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5574158390 0x7f55741fa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55740c6420 0x7f55741f86f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 --2- 172.21.15.32:0/2075125980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55740c6040 0x7f557414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 >> 172.21.15.32:0/2075125980 conn(0x7f557406c0d0 msgr2=0x7f55741500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 shutdown_connections 2024-03-16T20:02:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:28.255+0000 7f557bb7f640 1 -- 172.21.15.32:0/2075125980 wait complete. 2024-03-16T20:02:28.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:28.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:29.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 Processor -- start 2024-03-16T20:02:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 -- start start 2024-03-16T20:02:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f24c80c6420 0x7f24c8157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c8070760 con 0x7f24c806ba30 2024-03-16T20:02:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80731c0 con 0x7f24c80c6420 2024-03-16T20:02:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cebdb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80734a0 con 0x7f24c80c6040 2024-03-16T20:02:29.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24c7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f24c80c6420 0x7f24c8157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:29.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.987+0000 7f24cc977640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:29.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.991+0000 7f24cc977640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:56746/0 (socket says 172.21.15.32:56746) 2024-03-16T20:02:29.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.991+0000 7f24cc977640 1 -- 172.21.15.32:0/3114506741 learned_addr learned my addr 172.21.15.32:0/3114506741 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.995+0000 7f24cc977640 1 -- 172.21.15.32:0/3114506741 >> v2:172.21.15.59:3300/0 conn(0x7f24c80c6420 msgr2=0x7f24c8157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.995+0000 7f24cc977640 1 --2- 172.21.15.32:0/3114506741 >> v2:172.21.15.59:3300/0 conn(0x7f24c80c6420 0x7f24c8157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.995+0000 7f24cc977640 1 -- 172.21.15.32:0/3114506741 >> v1:172.21.15.59:6789/0 conn(0x7f24c806ba30 legacy=0x7f24c81516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.995+0000 7f24cc977640 1 -- 172.21.15.32:0/3114506741 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c806fe30 con 0x7f24c80c6040 2024-03-16T20:02:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:29.999+0000 7f24cc977640 1 --2- 172.21.15.32:0/3114506741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009870 tx=0x7f24bc01c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c5f41e8124375e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24c77fe640 1 -- 172.21.15.32:0/3114506741 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0ac620 con 0x7f24c80c6040 2024-03-16T20:02:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24c77fe640 1 -- 172.21.15.32:0/3114506741 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc0ac780 con 0x7f24c80c6040 2024-03-16T20:02:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24c77fe640 1 -- 172.21.15.32:0/3114506741 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0acb10 con 0x7f24c80c6040 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- 172.21.15.32:0/3114506741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 msgr2=0x7f24c80c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3114506741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009870 tx=0x7f24bc01c4f0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- 172.21.15.32:0/3114506741 shutdown_connections 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3114506741 >> v2:172.21.15.59:3300/0 conn(0x7f24c80c6420 0x7f24c8157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3114506741 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c80c8800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- 172.21.15.32:0/3114506741 >> 172.21.15.32:0/3114506741 conn(0x7f24c806c0d0 msgr2=0x7f24c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- 172.21.15.32:0/3114506741 shutdown_connections 2024-03-16T20:02:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- 172.21.15.32:0/3114506741 wait complete. 2024-03-16T20:02:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 Processor -- start 2024-03-16T20:02:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.011+0000 7f24cebdb640 1 -- start start 2024-03-16T20:02:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24c806ba30 0x7f24c8140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c81f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8149300 con 0x7f24c80c6420 2024-03-16T20:02:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c81f4e00 con 0x7f24c806ba30 2024-03-16T20:02:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cebdb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c813d040 con 0x7f24c80c6040 2024-03-16T20:02:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cd178640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24c7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c81f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cc977640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24c806ba30 0x7f24c8140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cd178640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45078/0 (socket says 172.21.15.32:45078) 2024-03-16T20:02:29.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.015+0000 7f24cd178640 1 -- 172.21.15.32:0/3054200100 learned_addr learned my addr 172.21.15.32:0/3054200100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.019+0000 7f24cd178640 1 -- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 msgr2=0x7f24c81f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.019+0000 7f24cd178640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c81f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.019+0000 7f24cd178640 1 -- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24c806ba30 msgr2=0x7f24c8140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.019+0000 7f24cd178640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24c806ba30 0x7f24c8140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.019+0000 7f24cd178640 1 -- 172.21.15.32:0/3054200100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0094d0 con 0x7f24c80c6420 2024-03-16T20:02:30.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.023+0000 7f24cd178640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c780 tx=0x7f24c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.027+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0003d90 con 0x7f24c80c6420 2024-03-16T20:02:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.027+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c000ce90 con 0x7f24c80c6420 2024-03-16T20:02:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.027+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c813d320 con 0x7f24c80c6420 2024-03-16T20:02:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.027+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00052b0 con 0x7f24c80c6420 2024-03-16T20:02:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.031+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c814a7c0 con 0x7f24c80c6420 2024-03-16T20:02:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.031+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c8148df0 con 0x7f24c80c6420 2024-03-16T20:02:30.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.067+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f24c0005410 con 0x7f24c80c6420 2024-03-16T20:02:30.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.067+0000 7f24c57fa640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 0x7f24ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:30.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.067+0000 7f24cc977640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 0x7f24ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:30.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.067+0000 7f24cc977640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 0x7f24ac069350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0ae000 tx=0x7f24bc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:30.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.371+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f24c0055120 con 0x7f24c80c6420 2024-03-16T20:02:30.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.371+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f24c00513c0 con 0x7f24c80c6420 2024-03-16T20:02:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.583+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24c8148df0 con 0x7f24c80c6420 2024-03-16T20:02:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24c57fa640 1 -- 172.21.15.32:0/3054200100 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f24c8148df0 con 0x7f24c80c6420 2024-03-16T20:02:30.584 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:30.584 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 msgr2=0x7f24ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 0x7f24ac069350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0ae000 tx=0x7f24bc0a7000 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 msgr2=0x7f24c813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c780 tx=0x7f24c000cc40 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 shutdown_connections 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f24ac066e90 0x7f24ac069350 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c80c6420 0x7f24c813b080 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24c80c6040 0x7f24c81f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 --2- 172.21.15.32:0/3054200100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24c806ba30 0x7f24c8140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 >> 172.21.15.32:0/3054200100 conn(0x7f24c806c0d0 msgr2=0x7f24c81566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.607+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 shutdown_connections 2024-03-16T20:02:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:30.611+0000 7f24cebdb640 1 -- 172.21.15.32:0/3054200100 wait complete. 2024-03-16T20:02:31.214 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:31.214 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:32.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.343+0000 7fcce006c640 1 Processor -- start 2024-03-16T20:02:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.343+0000 7fcce006c640 1 -- start start 2024-03-16T20:02:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fcce006c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccd8150f90 0x7fccd8156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fcce006c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd80c87f0 0x7fccd81616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fcce006c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd80700f0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fcce006c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd8072e70 con 0x7fccd8150f90 2024-03-16T20:02:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fcce006c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80728b0 con 0x7fccd80c87f0 2024-03-16T20:02:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fccdde08640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccd8150f90 0x7fccd8156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fccde609640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd80c87f0 0x7fccd81616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fccdde08640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccd8150f90 0x7fccd8156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45084/0 (socket says 172.21.15.32:45084) 2024-03-16T20:02:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.347+0000 7fccdde08640 1 -- 172.21.15.32:0/3859794899 learned_addr learned my addr 172.21.15.32:0/3859794899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.351+0000 7fccdde08640 1 -- 172.21.15.32:0/3859794899 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd80c87f0 msgr2=0x7fccd81616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:32.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.351+0000 7fccdde08640 1 --2- 172.21.15.32:0/3859794899 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd80c87f0 0x7fccd81616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.351+0000 7fccdde08640 1 -- 172.21.15.32:0/3859794899 >> v1:172.21.15.59:6789/0 conn(0x7fccd80c6fd0 legacy=0x7fccd806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.351+0000 7fccdde08640 1 -- 172.21.15.32:0/3859794899 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8072b90 con 0x7fccd8150f90 2024-03-16T20:02:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.355+0000 7fccdde08640 1 --2- 172.21.15.32:0/3859794899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd8150f90 0x7fccd8156d30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009870 tx=0x7fccd401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38987c3663a4889b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:32.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fccdce06640 1 -- 172.21.15.32:0/3859794899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40a9620 con 0x7fccd8150f90 2024-03-16T20:02:32.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fccdce06640 1 -- 172.21.15.32:0/3859794899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd40a9780 con 0x7fccd8150f90 2024-03-16T20:02:32.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fccdce06640 1 -- 172.21.15.32:0/3859794899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40a9b10 con 0x7fccd8150f90 2024-03-16T20:02:32.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- 172.21.15.32:0/3859794899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd8150f90 msgr2=0x7fccd8156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 --2- 172.21.15.32:0/3859794899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd8150f90 0x7fccd8156d30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009870 tx=0x7fccd401c560 comp rx=0 tx=0).stop 2024-03-16T20:02:32.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- 172.21.15.32:0/3859794899 shutdown_connections 2024-03-16T20:02:32.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 --2- 172.21.15.32:0/3859794899 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd80c87f0 0x7fccd81616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 --2- 172.21.15.32:0/3859794899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd8150f90 0x7fccd8156d30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- 172.21.15.32:0/3859794899 >> 172.21.15.32:0/3859794899 conn(0x7fccd80cc7a0 msgr2=0x7fccd80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- 172.21.15.32:0/3859794899 shutdown_connections 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- 172.21.15.32:0/3859794899 wait complete. 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 Processor -- start 2024-03-16T20:02:32.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.363+0000 7fcce006c640 1 -- start start 2024-03-16T20:02:32.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 0x7fccd81f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccd80c87f0 0x7fccd81451b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 0x7fccd81531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81548b0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81524f0 con 0x7fccd80c87f0 2024-03-16T20:02:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fcce006c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8151620 con 0x7fccd8150f90 2024-03-16T20:02:32.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fccdd607640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccd80c87f0 0x7fccd81451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fccde609640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 0x7fccd81531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fccde609640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 0x7fccd81531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:56790/0 (socket says 172.21.15.32:56790) 2024-03-16T20:02:32.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.367+0000 7fccde609640 1 -- 172.21.15.32:0/2096810513 learned_addr learned my addr 172.21.15.32:0/2096810513 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:32.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.371+0000 7fccdde08640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 0x7fccd81f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.375+0000 7fccdde08640 1 -- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 msgr2=0x7fccd81531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.375+0000 7fccdde08640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 0x7fccd81531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.375+0000 7fccdde08640 1 -- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccd80c87f0 msgr2=0x7fccd81451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.375+0000 7fccdde08640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccd80c87f0 0x7fccd81451b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.375+0000 7fccdde08640 1 -- 172.21.15.32:0/2096810513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40094d0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.379+0000 7fccdde08640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 0x7fccd81f3480 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccd40aa000 tx=0x7fccd4019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:32.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.383+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40b93f0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.383+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd40b9570 con 0x7fccd80c6fd0 2024-03-16T20:02:32.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.383+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd8157010 con 0x7fccd80c6fd0 2024-03-16T20:02:32.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.383+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40b9920 con 0x7fccd80c6fd0 2024-03-16T20:02:32.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.383+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd8151bc0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.387+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd8068000 con 0x7fccd80c6fd0 2024-03-16T20:02:32.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.419+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fccd40c3440 con 0x7fccd80c6fd0 2024-03-16T20:02:32.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.423+0000 7fccce7fc640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 0x7fccbc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:32.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.423+0000 7fccdd607640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 0x7fccbc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:32.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.423+0000 7fccdd607640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 0x7fccbc069350 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fccd815b0a0 tx=0x7fccc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:32.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.427+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fccd40ef950 con 0x7fccd80c6fd0 2024-03-16T20:02:32.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.735+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fccd40ebbf0 con 0x7fccd80c6fd0 2024-03-16T20:02:32.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:32.983+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccd80c3280 con 0x7fccd80c6fd0 2024-03-16T20:02:32.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.003+0000 7fccce7fc640 1 -- 172.21.15.32:0/2096810513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fccd40be090 con 0x7fccd80c6fd0 2024-03-16T20:02:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:32.984 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:32.984 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 msgr2=0x7fccbc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 0x7fccbc069350 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fccd815b0a0 tx=0x7fccc8009040 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 msgr2=0x7fccd81f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 0x7fccd81f3480 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccd40aa000 tx=0x7fccd4019300 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 shutdown_connections 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fccbc066e90 0x7fccbc069350 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fccd8150f90 0x7fccd81531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccd80c87f0 0x7fccd81451b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 --2- 172.21.15.32:0/2096810513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccd80c6fd0 0x7fccd81f3480 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 >> 172.21.15.32:0/2096810513 conn(0x7fccd80cc7a0 msgr2=0x7fccd80c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:32.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 shutdown_connections 2024-03-16T20:02:32.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:33.007+0000 7fcce006c640 1 -- 172.21.15.32:0/2096810513 wait complete. 2024-03-16T20:02:33.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:33.608 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:34.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 Processor -- start 2024-03-16T20:02:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 -- start start 2024-03-16T20:02:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38540c6420 0x7f38540c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3854158390 0x7f38541516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3854070760 con 0x7f38540c6040 2024-03-16T20:02:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38540731c0 con 0x7f3854158390 2024-03-16T20:02:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.739+0000 7f3859c2c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38540734a0 con 0x7f38540c6420 2024-03-16T20:02:34.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3854158390 0x7f38541516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:34.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3854158390 0x7f38541516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45106/0 (socket says 172.21.15.32:45106) 2024-03-16T20:02:34.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 -- 172.21.15.32:0/2319165930 learned_addr learned my addr 172.21.15.32:0/2319165930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3852ffd640 1 --2- 172.21.15.32:0/2319165930 >> v2:172.21.15.59:3300/0 conn(0x7f38540c6420 0x7f38540c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:34.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 -- 172.21.15.32:0/2319165930 >> v2:172.21.15.59:3300/0 conn(0x7f38540c6420 msgr2=0x7f38540c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:34.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 --2- 172.21.15.32:0/2319165930 >> v2:172.21.15.59:3300/0 conn(0x7f38540c6420 0x7f38540c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:34.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 -- 172.21.15.32:0/2319165930 >> v1:172.21.15.32:6789/0 conn(0x7f38540c6040 legacy=0x7f3854157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:34.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.743+0000 7f3853fff640 1 -- 172.21.15.32:0/2319165930 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385406fe30 con 0x7f3854158390 2024-03-16T20:02:34.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.747+0000 7f3853fff640 1 --2- 172.21.15.32:0/2319165930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3854158390 0x7f38541516d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f385406cf50 tx=0x7f384c01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcede22e1784a00d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.751+0000 7f38527fc640 1 -- 172.21.15.32:0/2319165930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0a5840 con 0x7f3854158390 2024-03-16T20:02:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.751+0000 7f38527fc640 1 -- 172.21.15.32:0/2319165930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c0a59a0 con 0x7f3854158390 2024-03-16T20:02:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.751+0000 7f38527fc640 1 -- 172.21.15.32:0/2319165930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0a5d30 con 0x7f3854158390 2024-03-16T20:02:34.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- 172.21.15.32:0/2319165930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3854158390 msgr2=0x7f38541516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:34.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- 172.21.15.32:0/2319165930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3854158390 0x7f38541516d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f385406cf50 tx=0x7f384c01c980 comp rx=0 tx=0).stop 2024-03-16T20:02:34.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- 172.21.15.32:0/2319165930 shutdown_connections 2024-03-16T20:02:34.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- 172.21.15.32:0/2319165930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3854158390 0x7f38541516d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- 172.21.15.32:0/2319165930 >> v2:172.21.15.59:3300/0 conn(0x7f38540c6420 0x7f38540c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- 172.21.15.32:0/2319165930 >> 172.21.15.32:0/2319165930 conn(0x7f385406c0d0 msgr2=0x7f385406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- 172.21.15.32:0/2319165930 shutdown_connections 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- 172.21.15.32:0/2319165930 wait complete. 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 Processor -- start 2024-03-16T20:02:34.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- start start 2024-03-16T20:02:34.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:34.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38540c6420 0x7f38541f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:34.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3854158390 0x7f38540cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3854157b50 con 0x7f38540c6040 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3854156de0 con 0x7f3854158390 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3859c2c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3854144920 con 0x7f38540c6420 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f38537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f38537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45120/0 (socket says 172.21.15.32:45120) 2024-03-16T20:02:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f38537fe640 1 -- 172.21.15.32:0/2540299999 learned_addr learned my addr 172.21.15.32:0/2540299999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:34.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.755+0000 7f3852ffd640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38540c6420 0x7f38541f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:34.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f3853fff640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3854158390 0x7f38540cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f38537fe640 1 -- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38540c6420 msgr2=0x7f38541f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f38537fe640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38540c6420 0x7f38541f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f38537fe640 1 -- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3854158390 msgr2=0x7f38540cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f38537fe640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3854158390 0x7f38540cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.759+0000 7f38537fe640 1 -- 172.21.15.32:0/2540299999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c009b30 con 0x7f38540c6040 2024-03-16T20:02:34.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.763+0000 7f38537fe640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f384400c490 tx=0x7f3844001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.767+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844003600 con 0x7f38540c6040 2024-03-16T20:02:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.767+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3844003780 con 0x7f38540c6040 2024-03-16T20:02:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.767+0000 7f3859c2c640 1 -- 172.21.15.32:0/2540299999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38541575e0 con 0x7f38540c6040 2024-03-16T20:02:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.767+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844003b30 con 0x7f38540c6040 2024-03-16T20:02:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.767+0000 7f3859c2c640 1 -- 172.21.15.32:0/2540299999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3854162060 con 0x7f38540c6040 2024-03-16T20:02:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:34.771+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c0052a0 con 0x7f38540c6040 2024-03-16T20:02:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.047+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3844003c90 con 0x7f38540c6040 2024-03-16T20:02:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.047+0000 7f383bfff640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.047+0000 7f3852ffd640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.047+0000 7f3852ffd640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 0x7f3840069300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3854156ce0 tx=0x7f3848008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.063+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f38440548e0 con 0x7f38540c6040 2024-03-16T20:02:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.063+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3844050b80 con 0x7f38540c6040 2024-03-16T20:02:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.391+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f382c005900 con 0x7f38540c6040 2024-03-16T20:02:35.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.415+0000 7f383bfff640 1 -- 172.21.15.32:0/2540299999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f382c005900 con 0x7f38540c6040 2024-03-16T20:02:35.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:35.395 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:35.395 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 msgr2=0x7f3840069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 0x7f3840069300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3854156ce0 tx=0x7f3848008040 comp rx=0 tx=0).stop 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 msgr2=0x7f38541570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f384400c490 tx=0x7f3844001e80 comp rx=0 tx=0).stop 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 shutdown_connections 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:35.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3854158390 0x7f38540cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38540c6420 0x7f38541f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 --2- 172.21.15.32:0/2540299999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38540c6040 0x7f38541570a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 >> 172.21.15.32:0/2540299999 conn(0x7f385406c0d0 msgr2=0x7f3854150510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 shutdown_connections 2024-03-16T20:02:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:35.419+0000 7f3839ffb640 1 -- 172.21.15.32:0/2540299999 wait complete. 2024-03-16T20:02:36.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:36.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:37.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.151+0000 7fae3e75c640 1 Processor -- start 2024-03-16T20:02:37.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.151+0000 7fae3e75c640 1 -- start start 2024-03-16T20:02:37.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3e75c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae38072d50 0x7fae3815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3e75c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 0x7fae380c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3e75c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae380c7c50 con 0x7fae38151ef0 2024-03-16T20:02:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3e75c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae380c70d0 con 0x7fae38072d50 2024-03-16T20:02:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3e75c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae38072a00 con 0x7fae380c7800 2024-03-16T20:02:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae377fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae38072d50 0x7fae3815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae377fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae38072d50 0x7fae3815b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45134/0 (socket says 172.21.15.32:45134) 2024-03-16T20:02:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae377fe640 1 -- 172.21.15.32:0/1190094847 learned_addr learned my addr 172.21.15.32:0/1190094847 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.155+0000 7fae3ccf9640 1 --2- 172.21.15.32:0/1190094847 >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 0x7fae380c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.159+0000 7fae377fe640 1 -- 172.21.15.32:0/1190094847 >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 msgr2=0x7fae380c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.159+0000 7fae377fe640 1 --2- 172.21.15.32:0/1190094847 >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 0x7fae380c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.159+0000 7fae377fe640 1 -- 172.21.15.32:0/1190094847 >> v1:172.21.15.59:6789/0 conn(0x7fae38151ef0 legacy=0x7fae380722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.159+0000 7fae377fe640 1 -- 172.21.15.32:0/1190094847 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae380c96b0 con 0x7fae38072d50 2024-03-16T20:02:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.159+0000 7fae3ccf9640 1 --2- 172.21.15.32:0/1190094847 >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 0x7fae380c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:02:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.163+0000 7fae377fe640 1 --2- 172.21.15.32:0/1190094847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae38072d50 0x7fae3815b2d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fae380c8df0 tx=0x7fae2c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1701cb7a92faf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae36ffd640 1 -- 172.21.15.32:0/1190094847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0a89d0 con 0x7fae38072d50 2024-03-16T20:02:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae36ffd640 1 -- 172.21.15.32:0/1190094847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c0a8b30 con 0x7fae38072d50 2024-03-16T20:02:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae36ffd640 1 -- 172.21.15.32:0/1190094847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0b0440 con 0x7fae38072d50 2024-03-16T20:02:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 -- 172.21.15.32:0/1190094847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae38072d50 msgr2=0x7fae3815b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1190094847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae38072d50 0x7fae3815b2d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fae380c8df0 tx=0x7fae2c01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 -- 172.21.15.32:0/1190094847 shutdown_connections 2024-03-16T20:02:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1190094847 >> v2:172.21.15.59:3300/0 conn(0x7fae380c7800 0x7fae380c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1190094847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae38072d50 0x7fae3815b2d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 -- 172.21.15.32:0/1190094847 >> 172.21.15.32:0/1190094847 conn(0x7fae3806e6c0 msgr2=0x7fae3806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 -- 172.21.15.32:0/1190094847 shutdown_connections 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.167+0000 7fae3e75c640 1 -- 172.21.15.32:0/1190094847 wait complete. 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 Processor -- start 2024-03-16T20:02:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 -- start start 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae38072d50 0x7fae38143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae38151ef0 0x7fae380ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38144f50 con 0x7fae380c7800 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38143cb0 con 0x7fae38072d50 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3e75c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae380ccc90 con 0x7fae38151ef0 2024-03-16T20:02:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae377fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae377fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45142/0 (socket says 172.21.15.32:45142) 2024-03-16T20:02:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae377fe640 1 -- 172.21.15.32:0/1939477625 learned_addr learned my addr 172.21.15.32:0/1939477625 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae3ccf9640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae38151ef0 0x7fae380ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.171+0000 7fae37fff640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae38072d50 0x7fae38143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.175+0000 7fae377fe640 1 -- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae38151ef0 msgr2=0x7fae380ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.175+0000 7fae377fe640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae38151ef0 0x7fae380ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.175+0000 7fae377fe640 1 -- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae38072d50 msgr2=0x7fae38143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.175+0000 7fae377fe640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae38072d50 0x7fae38143770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.175+0000 7fae377fe640 1 -- 172.21.15.32:0/1939477625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c0094d0 con 0x7fae380c7800 2024-03-16T20:02:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.179+0000 7fae377fe640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0b3040 tx=0x7fae2c01ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0b0e80 con 0x7fae380c7800 2024-03-16T20:02:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c0bb480 con 0x7fae380c7800 2024-03-16T20:02:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0bb810 con 0x7fae380c7800 2024-03-16T20:02:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae381451d0 con 0x7fae380c7800 2024-03-16T20:02:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae380cc750 con 0x7fae380c7800 2024-03-16T20:02:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.183+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae38068000 con 0x7fae380c7800 2024-03-16T20:02:37.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.471+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fae2c0bb970 con 0x7fae380c7800 2024-03-16T20:02:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.471+0000 7fae34ff9640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 0x7fae24069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.471+0000 7fae37fff640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 0x7fae24069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.471+0000 7fae37fff640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 0x7fae24069300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fae280098e0 tx=0x7fae28009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:37.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.495+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fae2c0f4680 con 0x7fae380c7800 2024-03-16T20:02:37.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.495+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fae2c0f0920 con 0x7fae380c7800 2024-03-16T20:02:37.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.783+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae38068000 con 0x7fae380c7800 2024-03-16T20:02:37.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.871+0000 7fae34ff9640 1 -- 172.21.15.32:0/1939477625 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fae2c0b5030 con 0x7fae380c7800 2024-03-16T20:02:37.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:37.852 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:37.852 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:37.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 msgr2=0x7fae24069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:37.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 0x7fae24069300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fae280098e0 tx=0x7fae28009060 comp rx=0 tx=0).stop 2024-03-16T20:02:37.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 msgr2=0x7fae381423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0b3040 tx=0x7fae2c01ba70 comp rx=0 tx=0).stop 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 shutdown_connections 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fae24066e40 0x7fae24069300 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae38151ef0 0x7fae380ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae380c7800 0x7fae381423f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 --2- 172.21.15.32:0/1939477625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae38072d50 0x7fae38143770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 >> 172.21.15.32:0/1939477625 conn(0x7fae3806e6c0 msgr2=0x7fae3814c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 shutdown_connections 2024-03-16T20:02:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:37.875+0000 7fae3e75c640 1 -- 172.21.15.32:0/1939477625 wait complete. 2024-03-16T20:02:38.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:38.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:39.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 Processor -- start 2024-03-16T20:02:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 -- start start 2024-03-16T20:02:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 0x7faa9c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c158c80 con 0x7faa9c0c97b0 2024-03-16T20:02:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c15bfc0 con 0x7faa9c15b2a0 2024-03-16T20:02:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faaa1e0f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0b0ad0 con 0x7faa9c151ef0 2024-03-16T20:02:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faa9affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faa9c0c97b0 0x7faa9c0c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44156/0 (socket says 172.21.15.32:44156) 2024-03-16T20:02:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faa9affd640 1 -- 172.21.15.32:0/3741903117 learned_addr learned my addr 172.21.15.32:0/3741903117 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.607+0000 7faa9b7fe640 1 --2- 172.21.15.32:0/3741903117 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 0x7faa9c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.611+0000 7faa9a7fc640 1 -- 172.21.15.32:0/3741903117 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152461336 0 0) 0x7faa9c158c80 con 0x7faa9c0c97b0 2024-03-16T20:02:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.611+0000 7faa9a7fc640 1 -- 172.21.15.32:0/3741903117 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa84003530 con 0x7faa9c0c97b0 2024-03-16T20:02:39.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.615+0000 7faa9b7fe640 1 -- 172.21.15.32:0/3741903117 >> v1:172.21.15.59:6789/0 conn(0x7faa9c15b2a0 legacy=0x7faa9c15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:39.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.615+0000 7faa9b7fe640 1 -- 172.21.15.32:0/3741903117 >> v1:172.21.15.32:6789/0 conn(0x7faa9c0c97b0 legacy=0x7faa9c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:39.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.615+0000 7faa9b7fe640 1 -- 172.21.15.32:0/3741903117 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c157ec0 con 0x7faa9c151ef0 2024-03-16T20:02:39.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.619+0000 7faa9b7fe640 1 --2- 172.21.15.32:0/3741903117 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 0x7faa9c0c9270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0097f0 tx=0x7faa8c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868dae0965070ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:39.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faa9a7fc640 1 -- 172.21.15.32:0/3741903117 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0a89d0 con 0x7faa9c151ef0 2024-03-16T20:02:39.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faa9a7fc640 1 -- 172.21.15.32:0/3741903117 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c0a8b30 con 0x7faa9c151ef0 2024-03-16T20:02:39.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faa9a7fc640 1 -- 172.21.15.32:0/3741903117 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0b0440 con 0x7faa9c151ef0 2024-03-16T20:02:39.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- 172.21.15.32:0/3741903117 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 msgr2=0x7faa9c0c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:39.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 --2- 172.21.15.32:0/3741903117 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 0x7faa9c0c9270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0097f0 tx=0x7faa8c01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- 172.21.15.32:0/3741903117 shutdown_connections 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 --2- 172.21.15.32:0/3741903117 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c151ef0 0x7faa9c0c9270 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- 172.21.15.32:0/3741903117 >> 172.21.15.32:0/3741903117 conn(0x7faa9c070f90 msgr2=0x7faa9c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- 172.21.15.32:0/3741903117 shutdown_connections 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- 172.21.15.32:0/3741903117 wait complete. 2024-03-16T20:02:39.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 Processor -- start 2024-03-16T20:02:39.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.631+0000 7faaa1e0f640 1 -- start start 2024-03-16T20:02:39.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c0c97b0 0x7faa9c1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:39.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa9c15b2a0 0x7faa9c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c15bde0 con 0x7faa9c151ef0 2024-03-16T20:02:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c1f3e50 con 0x7faa9c15b2a0 2024-03-16T20:02:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faaa1e0f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c14a1e0 con 0x7faa9c0c97b0 2024-03-16T20:02:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faa9affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faa9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa9c15b2a0 0x7faa9c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faa9affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33878/0 (socket says 172.21.15.32:33878) 2024-03-16T20:02:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faa9affd640 1 -- 172.21.15.32:0/1690780224 learned_addr learned my addr 172.21.15.32:0/1690780224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.635+0000 7faa9b7fe640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c0c97b0 0x7faa9c1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.639+0000 7faa9affd640 1 -- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c0c97b0 msgr2=0x7faa9c1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.639+0000 7faa9affd640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c0c97b0 0x7faa9c1f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.639+0000 7faa9affd640 1 -- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa9c15b2a0 msgr2=0x7faa9c06c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.639+0000 7faa9affd640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa9c15b2a0 0x7faa9c06c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.639+0000 7faa9affd640 1 -- 172.21.15.32:0/1690780224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c009420 con 0x7faa9c151ef0 2024-03-16T20:02:39.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.643+0000 7faa9affd640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c420 tx=0x7faa9000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90005cf0 con 0x7faa9c151ef0 2024-03-16T20:02:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa90005e70 con 0x7faa9c151ef0 2024-03-16T20:02:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faaa1e0f640 1 -- 172.21.15.32:0/1690780224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c14a4c0 con 0x7faa9c151ef0 2024-03-16T20:02:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa900072f0 con 0x7faa9c151ef0 2024-03-16T20:02:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faaa1e0f640 1 -- 172.21.15.32:0/1690780224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c1f3a70 con 0x7faa9c151ef0 2024-03-16T20:02:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.651+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c068000 con 0x7faa9c151ef0 2024-03-16T20:02:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.679+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faa90007470 con 0x7faa9c151ef0 2024-03-16T20:02:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.679+0000 7faa83fff640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 0x7faa88069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.679+0000 7faa9b7fe640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 0x7faa88069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:39.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.679+0000 7faa9b7fe640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 0x7faa88069350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0b2000 tx=0x7faa8c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.959+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7faa90007720 con 0x7faa9c151ef0 2024-03-16T20:02:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:39.959+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7faa90051430 con 0x7faa9c151ef0 2024-03-16T20:02:40.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.203+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa9c151720 con 0x7faa9c151ef0 2024-03-16T20:02:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.215+0000 7faa83fff640 1 -- 172.21.15.32:0/1690780224 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7faa90017070 con 0x7faa9c151ef0 2024-03-16T20:02:40.195 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:40.195 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 msgr2=0x7faa88069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 0x7faa88069350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0b2000 tx=0x7faa8c0aa070 comp rx=0 tx=0).stop 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 msgr2=0x7faa9c1f0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c420 tx=0x7faa9000c8e0 comp rx=0 tx=0).stop 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 shutdown_connections 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faa88066e90 0x7faa88069350 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa9c15b2a0 0x7faa9c06c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa9c151ef0 0x7faa9c1f0180 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 --2- 172.21.15.32:0/1690780224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faa9c0c97b0 0x7faa9c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 >> 172.21.15.32:0/1690780224 conn(0x7faa9c070f90 msgr2=0x7faa9c14b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 shutdown_connections 2024-03-16T20:02:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:40.219+0000 7faa81ffb640 1 -- 172.21.15.32:0/1690780224 wait complete. 2024-03-16T20:02:40.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:40.824 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:41.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:41.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 Processor -- start 2024-03-16T20:02:41.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 -- start start 2024-03-16T20:02:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85bc150d70 0x7f85bc073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc072dd0 con 0x7f85bc0725e0 2024-03-16T20:02:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc15d790 con 0x7f85bc15a420 2024-03-16T20:02:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.955+0000 7f85c460d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc15da50 con 0x7f85bc150d70 2024-03-16T20:02:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.959+0000 7f85c23a9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85bc150d70 0x7f85bc073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:41.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.959+0000 7f85c23a9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85bc150d70 0x7f85bc073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33890/0 (socket says 172.21.15.32:33890) 2024-03-16T20:02:41.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.959+0000 7f85c23a9640 1 -- 172.21.15.32:0/973631087 learned_addr learned my addr 172.21.15.32:0/973631087 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.963+0000 7f85c23a9640 1 -- 172.21.15.32:0/973631087 >> v1:172.21.15.59:6789/0 conn(0x7f85bc15a420 legacy=0x7f85bc15a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.963+0000 7f85c23a9640 1 -- 172.21.15.32:0/973631087 >> v1:172.21.15.32:6789/0 conn(0x7f85bc0725e0 legacy=0x7f85bc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.963+0000 7f85c23a9640 1 -- 172.21.15.32:0/973631087 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc15dd30 con 0x7f85bc150d70 2024-03-16T20:02:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.967+0000 7f85c23a9640 1 --2- 172.21.15.32:0/973631087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc150d70 0x7f85bc073070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009870 tx=0x7f85b801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d8e031b56acdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:41.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c13a7640 1 -- 172.21.15.32:0/973631087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80ac630 con 0x7f85bc150d70 2024-03-16T20:02:41.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c13a7640 1 -- 172.21.15.32:0/973631087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b80ac790 con 0x7f85bc150d70 2024-03-16T20:02:41.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c13a7640 1 -- 172.21.15.32:0/973631087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80acb20 con 0x7f85bc150d70 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 -- 172.21.15.32:0/973631087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc150d70 msgr2=0x7f85bc073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 --2- 172.21.15.32:0/973631087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc150d70 0x7f85bc073070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009870 tx=0x7f85b801c3d0 comp rx=0 tx=0).stop 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 -- 172.21.15.32:0/973631087 shutdown_connections 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 --2- 172.21.15.32:0/973631087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc150d70 0x7f85bc073070 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 -- 172.21.15.32:0/973631087 >> 172.21.15.32:0/973631087 conn(0x7f85bc06c0d0 msgr2=0x7f85bc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 -- 172.21.15.32:0/973631087 shutdown_connections 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.971+0000 7f85c460d640 1 -- 172.21.15.32:0/973631087 wait complete. 2024-03-16T20:02:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 Processor -- start 2024-03-16T20:02:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 -- start start 2024-03-16T20:02:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc0725e0 0x7f85bc144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc150d70 0x7f85bc1f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1f6510 con 0x7f85bc15a420 2024-03-16T20:02:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0cb880 con 0x7f85bc0725e0 2024-03-16T20:02:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c460d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0cbb40 con 0x7f85bc150d70 2024-03-16T20:02:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c2baa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c23a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc0725e0 0x7f85bc144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c1ba8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc150d70 0x7f85bc1f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c2baa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33894/0 (socket says 172.21.15.32:33894) 2024-03-16T20:02:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.975+0000 7f85c2baa640 1 -- 172.21.15.32:0/888432928 learned_addr learned my addr 172.21.15.32:0/888432928 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.983+0000 7f85c2baa640 1 -- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc150d70 msgr2=0x7f85bc1f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.983+0000 7f85c2baa640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc150d70 0x7f85bc1f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.983+0000 7f85c2baa640 1 -- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc0725e0 msgr2=0x7f85bc144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.983+0000 7f85c2baa640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc0725e0 0x7f85bc144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.983+0000 7f85c2baa640 1 -- 172.21.15.32:0/888432928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80094d0 con 0x7f85bc15a420 2024-03-16T20:02:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.987+0000 7f85c2baa640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c770 tx=0x7f85b400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003b80 con 0x7f85bc15a420 2024-03-16T20:02:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4003ce0 con 0x7f85bc15a420 2024-03-16T20:02:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc15d560 con 0x7f85bc15a420 2024-03-16T20:02:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b400aa90 con 0x7f85bc15a420 2024-03-16T20:02:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc0cc100 con 0x7f85bc15a420 2024-03-16T20:02:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:41.995+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc1f66c0 con 0x7f85bc15a420 2024-03-16T20:02:42.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:02:42.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:02:42.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.267+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85b400abf0 con 0x7f85bc15a420 2024-03-16T20:02:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.267+0000 7f85b2ffd640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.267+0000 7f85c23a9640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.271+0000 7f85c23a9640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f85b80ae000 tx=0x7f85b80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:42.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.291+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f85b4055c50 con 0x7f85bc15a420 2024-03-16T20:02:42.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.295+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f85b4051ef0 con 0x7f85bc15a420 2024-03-16T20:02:42.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.507+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85bc143a60 con 0x7f85bc15a420 2024-03-16T20:02:42.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85b2ffd640 1 -- 172.21.15.32:0/888432928 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f85b40555b0 con 0x7f85bc15a420 2024-03-16T20:02:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:42.509 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:42.509 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 msgr2=0x7f85a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f85b80ae000 tx=0x7f85b80a7000 comp rx=0 tx=0).stop 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 msgr2=0x7f85bc152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c770 tx=0x7f85b400cc30 comp rx=0 tx=0).stop 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 shutdown_connections 2024-03-16T20:02:42.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85a0066e90 0x7f85a0069350 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85bc15a420 0x7f85bc152ad0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85bc150d70 0x7f85bc1f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 --2- 172.21.15.32:0/888432928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc0725e0 0x7f85bc144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 >> 172.21.15.32:0/888432928 conn(0x7f85bc06c0d0 msgr2=0x7f85bc0c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 shutdown_connections 2024-03-16T20:02:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:42.531+0000 7f85c460d640 1 -- 172.21.15.32:0/888432928 wait complete. 2024-03-16T20:02:43.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:43.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:44.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.263+0000 7fe888269640 1 Processor -- start 2024-03-16T20:02:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.263+0000 7fe888269640 1 -- start start 2024-03-16T20:02:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe888269640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8800b1eb0 0x7fe880072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe888269640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8800c68e0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe888269640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe880067340 con 0x7fe880158980 2024-03-16T20:02:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe888269640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88006c1f0 con 0x7fe8800b1eb0 2024-03-16T20:02:44.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe886005640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8800b1eb0 0x7fe880072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:44.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe886005640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8800b1eb0 0x7fe880072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33900/0 (socket says 172.21.15.32:33900) 2024-03-16T20:02:44.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe886005640 1 -- 172.21.15.32:0/2028388335 learned_addr learned my addr 172.21.15.32:0/2028388335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.267+0000 7fe886806640 1 --1- 172.21.15.32:0/2028388335 >> v1:172.21.15.59:6789/0 conn(0x7fe880158980 0x7fe880162730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:58104/0 (socket says 172.21.15.32:58104) 2024-03-16T20:02:44.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.271+0000 7fe886005640 1 -- 172.21.15.32:0/2028388335 >> v1:172.21.15.59:6789/0 conn(0x7fe880158980 legacy=0x7fe880162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.271+0000 7fe886005640 1 -- 172.21.15.32:0/2028388335 >> v1:172.21.15.32:6789/0 conn(0x7fe8800c7ce0 legacy=0x7fe880158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:44.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.271+0000 7fe886005640 1 -- 172.21.15.32:0/2028388335 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88006c4d0 con 0x7fe8800b1eb0 2024-03-16T20:02:44.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.275+0000 7fe886005640 1 --2- 172.21.15.32:0/2028388335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800b1eb0 0x7fe880072a10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009870 tx=0x7fe87c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91197c357befa858 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:44.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe885003640 1 -- 172.21.15.32:0/2028388335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0a8a10 con 0x7fe8800b1eb0 2024-03-16T20:02:44.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe885003640 1 -- 172.21.15.32:0/2028388335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c0a8b70 con 0x7fe8800b1eb0 2024-03-16T20:02:44.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe885003640 1 -- 172.21.15.32:0/2028388335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0b0440 con 0x7fe8800b1eb0 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 -- 172.21.15.32:0/2028388335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800b1eb0 msgr2=0x7fe880072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 --2- 172.21.15.32:0/2028388335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800b1eb0 0x7fe880072a10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009870 tx=0x7fe87c01ed10 comp rx=0 tx=0).stop 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 -- 172.21.15.32:0/2028388335 shutdown_connections 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 --2- 172.21.15.32:0/2028388335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800b1eb0 0x7fe880072a10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 -- 172.21.15.32:0/2028388335 >> 172.21.15.32:0/2028388335 conn(0x7fe88006afc0 msgr2=0x7fe88006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 -- 172.21.15.32:0/2028388335 shutdown_connections 2024-03-16T20:02:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.279+0000 7fe888269640 1 -- 172.21.15.32:0/2028388335 wait complete. 2024-03-16T20:02:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 Processor -- start 2024-03-16T20:02:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 -- start start 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8800b1eb0 0x7fe8801f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 0x7fe8801572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 0x7fe880155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8801f0d70 con 0x7fe8800c7ce0 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8801f0780 con 0x7fe880158980 2024-03-16T20:02:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.283+0000 7fe888269640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe880146bf0 con 0x7fe8800b1eb0 2024-03-16T20:02:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.287+0000 7fe886806640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 0x7fe880155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.287+0000 7fe886806640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 0x7fe880155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:58690/0 (socket says 172.21.15.32:58690) 2024-03-16T20:02:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.287+0000 7fe886806640 1 -- 172.21.15.32:0/247142598 learned_addr learned my addr 172.21.15.32:0/247142598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.287+0000 7fe885804640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 0x7fe8801572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.287+0000 7fe886005640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8800b1eb0 0x7fe8801f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 -- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8800b1eb0 msgr2=0x7fe8801f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8800b1eb0 0x7fe8801f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 -- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 msgr2=0x7fe880155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 0x7fe880155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 -- 172.21.15.32:0/247142598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c0094d0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.291+0000 7fe885804640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 0x7fe8801572a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe8800c8150 tx=0x7fe87000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870003600 con 0x7fe8800c7ce0 2024-03-16T20:02:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe870003760 con 0x7fe8800c7ce0 2024-03-16T20:02:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8801f16d0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870003b30 con 0x7fe8800c7ce0 2024-03-16T20:02:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe880146630 con 0x7fe8800c7ce0 2024-03-16T20:02:44.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.299+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8801f0eb0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.603+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe870003c90 con 0x7fe8800c7ce0 2024-03-16T20:02:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.603+0000 7fe876ffd640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 0x7fe864069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.603+0000 7fe886005640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 0x7fe864069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.603+0000 7fe886005640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 0x7fe864069300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0b2000 tx=0x7fe87c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.619+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe8700551e0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.619+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe870051480 con 0x7fe8800c7ce0 2024-03-16T20:02:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.811+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8801f0eb0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.835+0000 7fe876ffd640 1 -- 172.21.15.32:0/247142598 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe8801f0eb0 con 0x7fe8800c7ce0 2024-03-16T20:02:44.815 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:44.815 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:44.816 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:44.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 msgr2=0x7fe864069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 0x7fe864069300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0b2000 tx=0x7fe87c008040 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 msgr2=0x7fe8801572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 0x7fe8801572a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe8800c8150 tx=0x7fe87000c870 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 shutdown_connections 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe864066e40 0x7fe864069300 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe880158980 0x7fe880155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8800c7ce0 0x7fe8801572a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 --2- 172.21.15.32:0/247142598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8800b1eb0 0x7fe8801f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 >> 172.21.15.32:0/247142598 conn(0x7fe88006afc0 msgr2=0x7fe88006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 shutdown_connections 2024-03-16T20:02:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:44.839+0000 7fe888269640 1 -- 172.21.15.32:0/247142598 wait complete. 2024-03-16T20:02:45.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:45.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:46.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:46.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 Processor -- start 2024-03-16T20:02:46.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 -- start start 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00154350 0x7f6f0006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f000c2d70 0x7f6f0015a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f00072a10 0x7f6f00072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0006bae0 con 0x7f6f000c2d70 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0015a540 con 0x7f6f00072a10 2024-03-16T20:02:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f07d0f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000c7fd0 con 0x7f6f00154350 2024-03-16T20:02:46.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f052aa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f000c2d70 0x7f6f0015a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.575+0000 7f6f05aab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00154350 0x7f6f0006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f062ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f00072a10 0x7f6f00072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f000c2d70 0x7f6f0015a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33932/0 (socket says 172.21.15.32:33932) 2024-03-16T20:02:46.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 -- 172.21.15.32:0/3917845761 learned_addr learned my addr 172.21.15.32:0/3917845761 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 -- 172.21.15.32:0/3917845761 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00154350 msgr2=0x7f6f0006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 --2- 172.21.15.32:0/3917845761 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00154350 0x7f6f0006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 -- 172.21.15.32:0/3917845761 >> v2:172.21.15.59:3300/0 conn(0x7f6f00072a10 msgr2=0x7f6f00072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 --2- 172.21.15.32:0/3917845761 >> v2:172.21.15.59:3300/0 conn(0x7f6f00072a10 0x7f6f00072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.579+0000 7f6f052aa640 1 -- 172.21.15.32:0/3917845761 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0006bd10 con 0x7f6f000c2d70 2024-03-16T20:02:46.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.583+0000 7f6f052aa640 1 --2- 172.21.15.32:0/3917845761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000c2d70 0x7f6f0015a9d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6f0006ca80 tx=0x7f6ef001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b6b37f98af39e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f04aa9640 1 -- 172.21.15.32:0/3917845761 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00ab600 con 0x7f6f000c2d70 2024-03-16T20:02:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f04aa9640 1 -- 172.21.15.32:0/3917845761 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00a7050 con 0x7f6f000c2d70 2024-03-16T20:02:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f04aa9640 1 -- 172.21.15.32:0/3917845761 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00ab980 con 0x7f6f000c2d70 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 -- 172.21.15.32:0/3917845761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000c2d70 msgr2=0x7f6f0015a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/3917845761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000c2d70 0x7f6f0015a9d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6f0006ca80 tx=0x7f6ef001c470 comp rx=0 tx=0).stop 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 -- 172.21.15.32:0/3917845761 shutdown_connections 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/3917845761 >> v2:172.21.15.59:3300/0 conn(0x7f6f00072a10 0x7f6f00072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/3917845761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000c2d70 0x7f6f0015a9d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/3917845761 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00154350 0x7f6f0006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 -- 172.21.15.32:0/3917845761 >> 172.21.15.32:0/3917845761 conn(0x7f6f0006afc0 msgr2=0x7f6f0006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 -- 172.21.15.32:0/3917845761 shutdown_connections 2024-03-16T20:02:46.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.587+0000 7f6f07d0f640 1 -- 172.21.15.32:0/3917845761 wait complete. 2024-03-16T20:02:46.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 Processor -- start 2024-03-16T20:02:46.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 -- start start 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00072a10 0x7f6f00152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f000c2d70 0x7f6f00156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001563e0 con 0x7f6f00154350 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00155e20 con 0x7f6f00072a10 2024-03-16T20:02:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f07d0f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00156100 con 0x7f6f000c2d70 2024-03-16T20:02:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f062ac640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f062ac640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33934/0 (socket says 172.21.15.32:33934) 2024-03-16T20:02:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f062ac640 1 -- 172.21.15.32:0/609958766 learned_addr learned my addr 172.21.15.32:0/609958766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.591+0000 7f6f052aa640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f000c2d70 0x7f6f00156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.595+0000 7f6f05aab640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00072a10 0x7f6f00152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.599+0000 7f6f062ac640 1 -- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f000c2d70 msgr2=0x7f6f00156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.599+0000 7f6f062ac640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f000c2d70 0x7f6f00156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.599+0000 7f6f062ac640 1 -- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00072a10 msgr2=0x7f6f00152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.599+0000 7f6f062ac640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00072a10 0x7f6f00152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.599+0000 7f6f062ac640 1 -- 172.21.15.32:0/609958766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef00094d0 con 0x7f6f00154350 2024-03-16T20:02:46.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.603+0000 7f6f062ac640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800eed0 tx=0x7f6ef800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.607+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8010040 con 0x7f6f00154350 2024-03-16T20:02:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.607+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f001505a0 con 0x7f6f00154350 2024-03-16T20:02:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.607+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f00150b60 con 0x7f6f00154350 2024-03-16T20:02:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.607+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef801d400 con 0x7f6f00154350 2024-03-16T20:02:46.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.607+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8009ea0 con 0x7f6f00154350 2024-03-16T20:02:46.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.611+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee80052a0 con 0x7f6f00154350 2024-03-16T20:02:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.883+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6ef8026420 con 0x7f6f00154350 2024-03-16T20:02:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.887+0000 7f6ef67fc640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 0x7f6ee4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.887+0000 7f6f05aab640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 0x7f6ee4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.887+0000 7f6f05aab640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 0x7f6ee4069300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009a50 tx=0x7f6efc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:46.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.911+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6ef8056ad0 con 0x7f6f00154350 2024-03-16T20:02:46.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:46.911+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6ef8052670 con 0x7f6f00154350 2024-03-16T20:02:47.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.167+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee8005900 con 0x7f6f00154350 2024-03-16T20:02:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.243+0000 7f6ef67fc640 1 -- 172.21.15.32:0/609958766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6ee8005900 con 0x7f6f00154350 2024-03-16T20:02:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:47.224 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:47.224 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 msgr2=0x7f6ee4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 0x7f6ee4069300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009a50 tx=0x7f6efc008040 comp rx=0 tx=0).stop 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 msgr2=0x7f6f0014e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800eed0 tx=0x7f6ef800b480 comp rx=0 tx=0).stop 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 shutdown_connections 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e40 0x7f6ee4069300 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00154350 0x7f6f0014e520 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:47.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f000c2d70 0x7f6f00156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:47.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 --2- 172.21.15.32:0/609958766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00072a10 0x7f6f00152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:47.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 >> 172.21.15.32:0/609958766 conn(0x7f6f0006afc0 msgr2=0x7f6f00066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:47.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 shutdown_connections 2024-03-16T20:02:47.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:47.247+0000 7f6f07d0f640 1 -- 172.21.15.32:0/609958766 wait complete. 2024-03-16T20:02:47.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:47.848 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:48.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.975+0000 7fe9c62b9640 1 Processor -- start 2024-03-16T20:02:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.975+0000 7fe9c62b9640 1 -- start start 2024-03-16T20:02:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c62b9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9c01540c0 0x7fe9c006f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c62b9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9c00c8550 0x7fe9c00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c62b9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c015ab60 con 0x7fe9c01544a0 2024-03-16T20:02:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c62b9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c006f910 con 0x7fe9c00c8550 2024-03-16T20:02:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c62b9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c0159fb0 con 0x7fe9c01540c0 2024-03-16T20:02:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9bf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe9c01544a0 0x7fe9c015b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44186/0 (socket says 172.21.15.32:44186) 2024-03-16T20:02:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9bf7fe640 1 -- 172.21.15.32:0/2055500606 learned_addr learned my addr 172.21.15.32:0/2055500606 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:48.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9bffff640 1 --2- 172.21.15.32:0/2055500606 >> v2:172.21.15.59:3300/0 conn(0x7fe9c01540c0 0x7fe9c006f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:48.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.979+0000 7fe9c4856640 1 --2- 172.21.15.32:0/2055500606 >> v2:172.21.15.32:3300/0 conn(0x7fe9c00c8550 0x7fe9c00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:48.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.987+0000 7fe9c4856640 1 -- 172.21.15.32:0/2055500606 >> v2:172.21.15.59:3300/0 conn(0x7fe9c01540c0 msgr2=0x7fe9c006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:48.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.987+0000 7fe9c4856640 1 --2- 172.21.15.32:0/2055500606 >> v2:172.21.15.59:3300/0 conn(0x7fe9c01540c0 0x7fe9c006f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:48.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.987+0000 7fe9c4856640 1 -- 172.21.15.32:0/2055500606 >> v1:172.21.15.32:6789/0 conn(0x7fe9c01544a0 legacy=0x7fe9c015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:48.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.987+0000 7fe9c4856640 1 -- 172.21.15.32:0/2055500606 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c006fb80 con 0x7fe9c00c8550 2024-03-16T20:02:48.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.991+0000 7fe9c4856640 1 --2- 172.21.15.32:0/2055500606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00cd830 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe9c006ee80 tx=0x7fe9b801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197b945f1aa61de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:48.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9beffd640 1 -- 172.21.15.32:0/2055500606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80a57d0 con 0x7fe9c00c8550 2024-03-16T20:02:48.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9beffd640 1 -- 172.21.15.32:0/2055500606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b80a5930 con 0x7fe9c00c8550 2024-03-16T20:02:48.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9beffd640 1 -- 172.21.15.32:0/2055500606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80a5cc0 con 0x7fe9c00c8550 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 -- 172.21.15.32:0/2055500606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 msgr2=0x7fe9c00cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/2055500606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00cd830 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe9c006ee80 tx=0x7fe9b801c930 comp rx=0 tx=0).stop 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 -- 172.21.15.32:0/2055500606 shutdown_connections 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/2055500606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00cd830 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/2055500606 >> v2:172.21.15.59:3300/0 conn(0x7fe9c01540c0 0x7fe9c006f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 -- 172.21.15.32:0/2055500606 >> 172.21.15.32:0/2055500606 conn(0x7fe9c0066ea0 msgr2=0x7fe9c00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 -- 172.21.15.32:0/2055500606 shutdown_connections 2024-03-16T20:02:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.995+0000 7fe9c62b9640 1 -- 172.21.15.32:0/2055500606 wait complete. 2024-03-16T20:02:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 Processor -- start 2024-03-16T20:02:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 -- start start 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c01540c0 0x7fe9c014e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c01544a0 0x7fe9c014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c01562c0 con 0x7fe9c00c8550 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00c3c60 con 0x7fe9c01544a0 2024-03-16T20:02:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c62b9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0156ce0 con 0x7fe9c01540c0 2024-03-16T20:02:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9bf7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c01540c0 0x7fe9c014e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9c4856640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c01544a0 0x7fe9c014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33946/0 (socket says 172.21.15.32:33946) 2024-03-16T20:02:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:48.999+0000 7fe9bffff640 1 -- 172.21.15.32:0/863073860 learned_addr learned my addr 172.21.15.32:0/863073860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:48.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 -- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c01540c0 msgr2=0x7fe9c014e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:48.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c01540c0 0x7fe9c014e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:48.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 -- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c01544a0 msgr2=0x7fe9c014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:48.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c01544a0 0x7fe9c014ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:48.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 -- 172.21.15.32:0/863073860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8009a70 con 0x7fe9c00c8550 2024-03-16T20:02:48.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.007+0000 7fe9bffff640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000e930 tx=0x7fe9b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0009690 con 0x7fe9c00c8550 2024-03-16T20:02:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b000ca00 con 0x7fe9c00c8550 2024-03-16T20:02:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c00c3980 con 0x7fe9c00c8550 2024-03-16T20:02:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000cd70 con 0x7fe9c00c8550 2024-03-16T20:02:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c01575b0 con 0x7fe9c00c8550 2024-03-16T20:02:48.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.015+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0157410 con 0x7fe9c00c8550 2024-03-16T20:02:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.051+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9b0013470 con 0x7fe9c00c8550 2024-03-16T20:02:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.051+0000 7fe9bcff9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 0x7fe9ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.051+0000 7fe9bf7fe640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 0x7fe9ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.055+0000 7fe9bf7fe640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 0x7fe9ac069350 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe9c015c690 tx=0x7fe9b4007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:49.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.059+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe9b0055a90 con 0x7fe9c00c8550 2024-03-16T20:02:49.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.339+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe9b0051d30 con 0x7fe9c00c8550 2024-03-16T20:02:49.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.547+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c0157410 con 0x7fe9c00c8550 2024-03-16T20:02:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.571+0000 7fe9bcff9640 1 -- 172.21.15.32:0/863073860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe9c0157410 con 0x7fe9c00c8550 2024-03-16T20:02:49.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:49.551 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:49.551 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 msgr2=0x7fe9ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 0x7fe9ac069350 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe9c015c690 tx=0x7fe9b4007e90 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 msgr2=0x7fe9c00c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000e930 tx=0x7fe9b000edf0 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 shutdown_connections 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9ac066e90 0x7fe9ac069350 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c01544a0 0x7fe9c014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c01540c0 0x7fe9c014e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 --2- 172.21.15.32:0/863073860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c00c8550 0x7fe9c00c5e80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 >> 172.21.15.32:0/863073860 conn(0x7fe9c0066ea0 msgr2=0x7fe9c015d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 shutdown_connections 2024-03-16T20:02:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:49.575+0000 7fe9c62b9640 1 -- 172.21.15.32:0/863073860 wait complete. 2024-03-16T20:02:50.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:50.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:51.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 Processor -- start 2024-03-16T20:02:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 -- start start 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fafc40c7470 0x7fafc4165780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc4070c90 con 0x7fafc4153fa0 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc40c8590 con 0x7fafc4070840 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafcbd79640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc40c3f00 con 0x7fafc40c7470 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafc9b15640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fafc4153fa0 0x7fafc415d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52910/0 (socket says 172.21.15.32:52910) 2024-03-16T20:02:51.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafc9b15640 1 -- 172.21.15.32:0/2370708039 learned_addr learned my addr 172.21.15.32:0/2370708039 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:02:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.307+0000 7fafca316640 1 --2- 172.21.15.32:0/2370708039 >> v2:172.21.15.59:3300/0 conn(0x7fafc40c7470 0x7fafc4165780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.311+0000 7fafc8b13640 1 -- 172.21.15.32:0/2370708039 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075526251 0 0) 0x7fafc0001120 con 0x7fafc4153fa0 2024-03-16T20:02:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.311+0000 7fafc8b13640 1 -- 172.21.15.32:0/2370708039 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb0003530 con 0x7fafc4153fa0 2024-03-16T20:02:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.315+0000 7fafca316640 1 -- 172.21.15.32:0/2370708039 >> v1:172.21.15.59:6789/0 conn(0x7fafc4070840 legacy=0x7fafc40724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.315+0000 7fafca316640 1 -- 172.21.15.32:0/2370708039 >> v1:172.21.15.32:6789/0 conn(0x7fafc4153fa0 legacy=0x7fafc415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.315+0000 7fafca316640 1 -- 172.21.15.32:0/2370708039 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4073590 con 0x7fafc40c7470 2024-03-16T20:02:51.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.319+0000 7fafca316640 1 --2- 172.21.15.32:0/2370708039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc40c7470 0x7fafc4165780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009d90 tx=0x7fafbc01f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79cbbf2977b943ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.323+0000 7fafc8b13640 1 -- 172.21.15.32:0/2370708039 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc0af600 con 0x7fafc40c7470 2024-03-16T20:02:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafc8b13640 1 -- 172.21.15.32:0/2370708039 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc0ab050 con 0x7fafc40c7470 2024-03-16T20:02:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafc8b13640 1 -- 172.21.15.32:0/2370708039 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc0af960 con 0x7fafc40c7470 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 -- 172.21.15.32:0/2370708039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc40c7470 msgr2=0x7fafc4165780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 --2- 172.21.15.32:0/2370708039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc40c7470 0x7fafc4165780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009d90 tx=0x7fafbc01f120 comp rx=0 tx=0).stop 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 -- 172.21.15.32:0/2370708039 shutdown_connections 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 --2- 172.21.15.32:0/2370708039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc40c7470 0x7fafc4165780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 -- 172.21.15.32:0/2370708039 >> 172.21.15.32:0/2370708039 conn(0x7fafc406c0d0 msgr2=0x7fafc406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 -- 172.21.15.32:0/2370708039 shutdown_connections 2024-03-16T20:02:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.327+0000 7fafcbd79640 1 -- 172.21.15.32:0/2370708039 wait complete. 2024-03-16T20:02:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 Processor -- start 2024-03-16T20:02:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 -- start start 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc4070840 0x7fafc4146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fafc4153fa0 0x7fafc4154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4153680 con 0x7fafc40c7470 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc41f0c60 con 0x7fafc4070840 2024-03-16T20:02:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafcbd79640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc41efe00 con 0x7fafc4153fa0 2024-03-16T20:02:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafc9314640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafca316640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fafc4153fa0 0x7fafc4154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafc9314640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51576/0 (socket says 172.21.15.32:51576) 2024-03-16T20:02:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafc9314640 1 -- 172.21.15.32:0/2373653329 learned_addr learned my addr 172.21.15.32:0/2373653329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.331+0000 7fafc9b15640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc4070840 0x7fafc4146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:51.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 -- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fafc4153fa0 msgr2=0x7fafc4154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fafc4153fa0 0x7fafc4154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 -- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc4070840 msgr2=0x7fafc4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc4070840 0x7fafc4146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 -- 172.21.15.32:0/2373653329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc009a70 con 0x7fafc40c7470 2024-03-16T20:02:51.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.339+0000 7fafc9314640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c490 tx=0x7fafb4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4003600 con 0x7fafc40c7470 2024-03-16T20:02:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb4003780 con 0x7fafc40c7470 2024-03-16T20:02:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4003b70 con 0x7fafc40c7470 2024-03-16T20:02:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafcbd79640 1 -- 172.21.15.32:0/2373653329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc41f00e0 con 0x7fafc40c7470 2024-03-16T20:02:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafcbd79640 1 -- 172.21.15.32:0/2373653329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc41f06a0 con 0x7fafc40c7470 2024-03-16T20:02:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.347+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa00052a0 con 0x7fafc40c7470 2024-03-16T20:02:51.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.643+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fafb4003cd0 con 0x7fafc40c7470 2024-03-16T20:02:51.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.643+0000 7fafba7fc640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 0x7fafa8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:51.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.643+0000 7fafc9b15640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 0x7fafa8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.643+0000 7fafc9b15640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 0x7fafa8069350 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fafc0007fd0 tx=0x7fafc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.659+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fafb4055030 con 0x7fafc40c7470 2024-03-16T20:02:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.659+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fafb40512d0 con 0x7fafc40c7470 2024-03-16T20:02:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.851+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa0005900 con 0x7fafc40c7470 2024-03-16T20:02:51.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.871+0000 7fafba7fc640 1 -- 172.21.15.32:0/2373653329 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fafa0005900 con 0x7fafc40c7470 2024-03-16T20:02:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:51.852 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 msgr2=0x7fafa8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 0x7fafa8069350 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fafc0007fd0 tx=0x7fafc0009040 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 msgr2=0x7fafc40ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c490 tx=0x7fafb4001e80 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 shutdown_connections 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fafa8066e90 0x7fafa8069350 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fafc4153fa0 0x7fafc4154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc40c7470 0x7fafc40ccd10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 --2- 172.21.15.32:0/2373653329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc4070840 0x7fafc4146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 >> 172.21.15.32:0/2373653329 conn(0x7fafc406c0d0 msgr2=0x7fafc4151310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 shutdown_connections 2024-03-16T20:02:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:51.875+0000 7fafa7fff640 1 -- 172.21.15.32:0/2373653329 wait complete. 2024-03-16T20:02:52.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:52.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:53.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.611+0000 7f5cc71bd640 1 Processor -- start 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 -- start start 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5cc00cc590 0x7f5cc00cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5cc015dbb0 0x7f5cc00ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc00cbfa0 con 0x7f5cc00caab0 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc00cceb0 con 0x7f5cc015dbb0 2024-03-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc71bd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc015a140 con 0x7f5cc00cc590 2024-03-16T20:02:53.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cbffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5cc015dbb0 0x7f5cc00ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cc4f59640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5cc00cc590 0x7f5cc00cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cbffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5cc015dbb0 0x7f5cc00ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51590/0 (socket says 172.21.15.32:51590) 2024-03-16T20:02:53.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.615+0000 7f5cbffff640 1 -- 172.21.15.32:0/1736664314 learned_addr learned my addr 172.21.15.32:0/1736664314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:53.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.619+0000 7f5cbffff640 1 -- 172.21.15.32:0/1736664314 >> v2:172.21.15.59:3300/0 conn(0x7f5cc00cc590 msgr2=0x7f5cc00cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:53.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.619+0000 7f5cbffff640 1 --2- 172.21.15.32:0/1736664314 >> v2:172.21.15.59:3300/0 conn(0x7f5cc00cc590 0x7f5cc00cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.619+0000 7f5cbffff640 1 -- 172.21.15.32:0/1736664314 >> v1:172.21.15.59:6789/0 conn(0x7f5cc00caab0 legacy=0x7f5cc006bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:53.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.619+0000 7f5cbffff640 1 -- 172.21.15.32:0/1736664314 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc015a3e0 con 0x7f5cc015dbb0 2024-03-16T20:02:53.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.623+0000 7f5cbffff640 1 --2- 172.21.15.32:0/1736664314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc015dbb0 0x7f5cc00ca570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00ccff0 tx=0x7f5cac01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767cb84b125d6876 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cbf7fe640 1 -- 172.21.15.32:0/1736664314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac0a89d0 con 0x7f5cc015dbb0 2024-03-16T20:02:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cbf7fe640 1 -- 172.21.15.32:0/1736664314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cac0a8b30 con 0x7f5cc015dbb0 2024-03-16T20:02:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cbf7fe640 1 -- 172.21.15.32:0/1736664314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac0b0440 con 0x7f5cc015dbb0 2024-03-16T20:02:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 -- 172.21.15.32:0/1736664314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc015dbb0 msgr2=0x7f5cc00ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/1736664314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc015dbb0 0x7f5cc00ca570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00ccff0 tx=0x7f5cac01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 -- 172.21.15.32:0/1736664314 shutdown_connections 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/1736664314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc015dbb0 0x7f5cc00ca570 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/1736664314 >> v2:172.21.15.59:3300/0 conn(0x7f5cc00cc590 0x7f5cc00cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 -- 172.21.15.32:0/1736664314 >> 172.21.15.32:0/1736664314 conn(0x7f5cc0066ea0 msgr2=0x7f5cc00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 -- 172.21.15.32:0/1736664314 shutdown_connections 2024-03-16T20:02:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.627+0000 7f5cc71bd640 1 -- 172.21.15.32:0/1736664314 wait complete. 2024-03-16T20:02:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 Processor -- start 2024-03-16T20:02:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 -- start start 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5cc00cc590 0x7f5cc01efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc015dbb0 0x7f5cc01f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc0145eb0 con 0x7f5cc00caab0 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00c4830 con 0x7f5cc015dbb0 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc71bd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00c5950 con 0x7f5cc00cc590 2024-03-16T20:02:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc4f59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc4f59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51592/0 (socket says 172.21.15.32:51592) 2024-03-16T20:02:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc4f59640 1 -- 172.21.15.32:0/409109648 learned_addr learned my addr 172.21.15.32:0/409109648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cc575a640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc015dbb0 0x7f5cc01f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.631+0000 7f5cbffff640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5cc00cc590 0x7f5cc01efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.635+0000 7f5cc4f59640 1 -- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5cc00cc590 msgr2=0x7f5cc01efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.635+0000 7f5cc4f59640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5cc00cc590 0x7f5cc01efb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.635+0000 7f5cc4f59640 1 -- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc015dbb0 msgr2=0x7f5cc01f00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.635+0000 7f5cc4f59640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc015dbb0 0x7f5cc01f00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.635+0000 7f5cc4f59640 1 -- 172.21.15.32:0/409109648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac0094d0 con 0x7f5cc00caab0 2024-03-16T20:02:53.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.639+0000 7f5cc4f59640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400e930 tx=0x7f5cb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:53.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4009690 con 0x7f5cc00caab0 2024-03-16T20:02:53.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb400ca00 con 0x7f5cc00caab0 2024-03-16T20:02:53.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb400cd70 con 0x7f5cc00caab0 2024-03-16T20:02:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc014a950 con 0x7f5cc00caab0 2024-03-16T20:02:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc00c6520 con 0x7f5cc00caab0 2024-03-16T20:02:53.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.643+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc006a340 con 0x7f5cc00caab0 2024-03-16T20:02:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.911+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5cb4013470 con 0x7f5cc00caab0 2024-03-16T20:02:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.911+0000 7f5cbd7fa640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 0x7f5ca4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.911+0000 7f5cbffff640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 0x7f5ca4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:53.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.911+0000 7f5cbffff640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 0x7f5ca4069350 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0b2000 tx=0x7f5cac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:53.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.939+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f5cb4056100 con 0x7f5cc00caab0 2024-03-16T20:02:53.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:53.939+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5cb40523a0 con 0x7f5cc00caab0 2024-03-16T20:02:54.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.171+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cc01ef790 con 0x7f5cc00caab0 2024-03-16T20:02:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.207+0000 7f5cbd7fa640 1 -- 172.21.15.32:0/409109648 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f5cb4021020 con 0x7f5cc00caab0 2024-03-16T20:02:54.186 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:54.186 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:54.186 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 msgr2=0x7f5ca4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 0x7f5ca4069350 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0b2000 tx=0x7f5cac009040 comp rx=0 tx=0).stop 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 msgr2=0x7f5cc01595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400e930 tx=0x7f5cb400edf0 comp rx=0 tx=0).stop 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 shutdown_connections 2024-03-16T20:02:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5ca4066e90 0x7f5ca4069350 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc015dbb0 0x7f5cc01f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5cc00cc590 0x7f5cc01efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 --2- 172.21.15.32:0/409109648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc00caab0 0x7f5cc01595f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 >> 172.21.15.32:0/409109648 conn(0x7f5cc0066ea0 msgr2=0x7f5cc00cdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 shutdown_connections 2024-03-16T20:02:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:54.211+0000 7f5cc71bd640 1 -- 172.21.15.32:0/409109648 wait complete. 2024-03-16T20:02:54.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:54.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:55.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:55.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 Processor -- start 2024-03-16T20:02:55.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 -- start start 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f418806de40 0x7f418815a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41880c73f0 0x7f4188153e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4188071b70 con 0x7f41880c73f0 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418815a390 con 0x7f418806de40 2024-03-16T20:02:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f418d69e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4188072bd0 con 0x7f418806e220 2024-03-16T20:02:55.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f41867fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41880c73f0 0x7f4188153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f41877fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f4186ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f418806de40 0x7f418815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f41877fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:40690/0 (socket says 172.21.15.32:40690) 2024-03-16T20:02:55.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.939+0000 7f41877fe640 1 -- 172.21.15.32:0/1779967185 learned_addr learned my addr 172.21.15.32:0/1779967185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.943+0000 7f41877fe640 1 -- 172.21.15.32:0/1779967185 >> v2:172.21.15.59:3300/0 conn(0x7f418806de40 msgr2=0x7f418815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.943+0000 7f41877fe640 1 --2- 172.21.15.32:0/1779967185 >> v2:172.21.15.59:3300/0 conn(0x7f418806de40 0x7f418815a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.943+0000 7f41877fe640 1 -- 172.21.15.32:0/1779967185 >> v2:172.21.15.32:3300/0 conn(0x7f41880c73f0 msgr2=0x7f4188153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.943+0000 7f41877fe640 1 --2- 172.21.15.32:0/1779967185 >> v2:172.21.15.32:3300/0 conn(0x7f41880c73f0 0x7f4188153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.943+0000 7f41877fe640 1 -- 172.21.15.32:0/1779967185 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188072eb0 con 0x7f418806e220 2024-03-16T20:02:55.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.951+0000 7f41877fe640 1 --2- 172.21.15.32:0/1779967185 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41880cd020 tx=0x7f418001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61613447e3db5f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:55.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f4185ffb640 1 -- 172.21.15.32:0/1779967185 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41800af400 con 0x7f418806e220 2024-03-16T20:02:55.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f4185ffb640 1 -- 172.21.15.32:0/1779967185 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41800ab050 con 0x7f418806e220 2024-03-16T20:02:55.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f4185ffb640 1 -- 172.21.15.32:0/1779967185 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41800af780 con 0x7f418806e220 2024-03-16T20:02:55.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- 172.21.15.32:0/1779967185 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 msgr2=0x7f4188154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:55.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 --2- 172.21.15.32:0/1779967185 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41880cd020 tx=0x7f418001f1e0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- 172.21.15.32:0/1779967185 shutdown_connections 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 --2- 172.21.15.32:0/1779967185 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f4188154390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 --2- 172.21.15.32:0/1779967185 >> v2:172.21.15.32:3300/0 conn(0x7f41880c73f0 0x7f4188153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 --2- 172.21.15.32:0/1779967185 >> v2:172.21.15.59:3300/0 conn(0x7f418806de40 0x7f418815a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- 172.21.15.32:0/1779967185 >> 172.21.15.32:0/1779967185 conn(0x7f41880cbfe0 msgr2=0x7f41880cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- 172.21.15.32:0/1779967185 shutdown_connections 2024-03-16T20:02:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- 172.21.15.32:0/1779967185 wait complete. 2024-03-16T20:02:55.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 Processor -- start 2024-03-16T20:02:55.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.963+0000 7f418d69e640 1 -- start start 2024-03-16T20:02:55.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f41880c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41880c48d0 0x7f41880c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41881efeb0 con 0x7f418806de40 2024-03-16T20:02:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41880c3f00 con 0x7f41880c48d0 2024-03-16T20:02:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f418d69e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41880c4d80 con 0x7f418806e220 2024-03-16T20:02:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f4186ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f41877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41880c48d0 0x7f41880c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f41867fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f41880c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f4186ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51616/0 (socket says 172.21.15.32:51616) 2024-03-16T20:02:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.967+0000 7f4186ffd640 1 -- 172.21.15.32:0/14716213 learned_addr learned my addr 172.21.15.32:0/14716213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 -- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 msgr2=0x7f41880c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f41880c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 -- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41880c48d0 msgr2=0x7f41880c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41880c48d0 0x7f41880c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 -- 172.21.15.32:0/14716213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4180009b30 con 0x7f418806de40 2024-03-16T20:02:55.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.971+0000 7f4186ffd640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f417800e870 tx=0x7f417800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:55.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800ee60 con 0x7f418806de40 2024-03-16T20:02:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417800c940 con 0x7f418806de40 2024-03-16T20:02:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800ccf0 con 0x7f418806de40 2024-03-16T20:02:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41881f0380 con 0x7f418806de40 2024-03-16T20:02:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41880c38b0 con 0x7f418806de40 2024-03-16T20:02:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:55.979+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41880674e0 con 0x7f418806de40 2024-03-16T20:02:55.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.019+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4178013470 con 0x7f418806de40 2024-03-16T20:02:55.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.019+0000 7f416f7fe640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 0x7f4174069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:55.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.019+0000 7f41867fc640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 0x7f4174069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:55.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.019+0000 7f41867fc640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 0x7f4174069300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f41880c9860 tx=0x7f417c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.019+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4178055df0 con 0x7f418806de40 2024-03-16T20:02:56.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.319+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4178052090 con 0x7f418806de40 2024-03-16T20:02:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.531+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4188068000 con 0x7f418806de40 2024-03-16T20:02:56.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.555+0000 7f416f7fe640 1 -- 172.21.15.32:0/14716213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4178017090 con 0x7f418806de40 2024-03-16T20:02:56.535 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:56.535 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:56.535 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 msgr2=0x7f4174069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 0x7f4174069300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f41880c9860 tx=0x7f417c009190 comp rx=0 tx=0).stop 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 msgr2=0x7f41880c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f417800e870 tx=0x7f417800ed30 comp rx=0 tx=0).stop 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 shutdown_connections 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4174066e40 0x7f4174069300 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41880c48d0 0x7f41880c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f418806e220 0x7f41880c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 --2- 172.21.15.32:0/14716213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f418806de40 0x7f41880c22b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 >> 172.21.15.32:0/14716213 conn(0x7f41880cbfe0 msgr2=0x7f418806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 shutdown_connections 2024-03-16T20:02:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:56.559+0000 7f418d69e640 1 -- 172.21.15.32:0/14716213 wait complete. 2024-03-16T20:02:57.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:57.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:02:58.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:02:58.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 Processor -- start 2024-03-16T20:02:58.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 -- start start 2024-03-16T20:02:58.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f2390156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23900c87f0 0x7f23901616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23900700f0 con 0x7f23900c6fd0 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2390072e70 con 0x7f23900c87f0 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f2395ef1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23900728b0 con 0x7f2390150f90 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f238ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23900c87f0 0x7f23901616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f238f7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f2390156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f238ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23900c87f0 0x7f23901616b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51620/0 (socket says 172.21.15.32:51620) 2024-03-16T20:02:58.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.303+0000 7f238ffff640 1 -- 172.21.15.32:0/329895930 learned_addr learned my addr 172.21.15.32:0/329895930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.307+0000 7f238ffff640 1 -- 172.21.15.32:0/329895930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 msgr2=0x7f2390156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.307+0000 7f238ffff640 1 --2- 172.21.15.32:0/329895930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f2390156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.307+0000 7f238ffff640 1 -- 172.21.15.32:0/329895930 >> v1:172.21.15.32:6789/0 conn(0x7f23900c6fd0 legacy=0x7f239006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:02:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.307+0000 7f238ffff640 1 -- 172.21.15.32:0/329895930 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390072b90 con 0x7f23900c87f0 2024-03-16T20:02:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.311+0000 7f238ffff640 1 --2- 172.21.15.32:0/329895930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f23901616b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2388009e50 tx=0x7f238801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b57ed5d65e3e06a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.315+0000 7f238e7fc640 1 -- 172.21.15.32:0/329895930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880a5870 con 0x7f23900c87f0 2024-03-16T20:02:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.315+0000 7f238e7fc640 1 -- 172.21.15.32:0/329895930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23880a59d0 con 0x7f23900c87f0 2024-03-16T20:02:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.315+0000 7f238e7fc640 1 -- 172.21.15.32:0/329895930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880a5d60 con 0x7f23900c87f0 2024-03-16T20:02:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- 172.21.15.32:0/329895930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 msgr2=0x7f23901616b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- 172.21.15.32:0/329895930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f23901616b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2388009e50 tx=0x7f238801c9f0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- 172.21.15.32:0/329895930 shutdown_connections 2024-03-16T20:02:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- 172.21.15.32:0/329895930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f23901616b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- 172.21.15.32:0/329895930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f2390156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- 172.21.15.32:0/329895930 >> 172.21.15.32:0/329895930 conn(0x7f23900cc7a0 msgr2=0x7f23900ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- 172.21.15.32:0/329895930 shutdown_connections 2024-03-16T20:02:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- 172.21.15.32:0/329895930 wait complete. 2024-03-16T20:02:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 Processor -- start 2024-03-16T20:02:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- start start 2024-03-16T20:02:58.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23900c6fd0 0x7f23901648b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f239014b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239014c6e0 con 0x7f23900c87f0 2024-03-16T20:02:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23901f3de0 con 0x7f23900c6fd0 2024-03-16T20:02:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.319+0000 7f2395ef1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239015f860 con 0x7f2390150f90 2024-03-16T20:02:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.323+0000 7f238effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.323+0000 7f238ffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f239014b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.323+0000 7f238effd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51626/0 (socket says 172.21.15.32:51626) 2024-03-16T20:02:58.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.323+0000 7f238effd640 1 -- 172.21.15.32:0/3603546776 learned_addr learned my addr 172.21.15.32:0/3603546776 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:02:58.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.323+0000 7f238f7fe640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23900c6fd0 0x7f23901648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 -- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 msgr2=0x7f239014b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f239014b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 -- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23900c6fd0 msgr2=0x7f23901648b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23900c6fd0 0x7f23901648b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 -- 172.21.15.32:0/3603546776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388009b30 con 0x7f23900c87f0 2024-03-16T20:02:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.327+0000 7f238effd640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f238400b3e0 tx=0x7f238400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.331+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384011830 con 0x7f23900c87f0 2024-03-16T20:02:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.331+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239015fb40 con 0x7f23900c87f0 2024-03-16T20:02:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.331+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239014aff0 con 0x7f23900c87f0 2024-03-16T20:02:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.331+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2390142430 con 0x7f23900c87f0 2024-03-16T20:02:58.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.335+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384009210 con 0x7f23900c87f0 2024-03-16T20:02:58.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.335+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384011b90 con 0x7f23900c87f0 2024-03-16T20:02:58.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.655+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f238401a400 con 0x7f23900c87f0 2024-03-16T20:02:58.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.655+0000 7f2377fff640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 0x7f237c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:02:58.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.655+0000 7f238f7fe640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 0x7f237c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:02:58.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.655+0000 7f238f7fe640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 0x7f237c069300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2380009b60 tx=0x7f2380009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:02:58.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.679+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2384055ce0 con 0x7f23900c87f0 2024-03-16T20:02:58.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.679+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2384055ab0 con 0x7f23900c87f0 2024-03-16T20:02:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.895+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23900c3280 con 0x7f23900c87f0 2024-03-16T20:02:58.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.991+0000 7f2377fff640 1 -- 172.21.15.32:0/3603546776 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f2384015070 con 0x7f23900c87f0 2024-03-16T20:02:58.972 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:02:58.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:02:58.973 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:02:58.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 msgr2=0x7f237c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:58.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 0x7f237c069300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2380009b60 tx=0x7f2380009190 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 msgr2=0x7f239014f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f238400b3e0 tx=0x7f238400b8a0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 shutdown_connections 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f237c066e40 0x7f237c069300 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2390150f90 0x7f239014b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23900c87f0 0x7f239014f3b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 --2- 172.21.15.32:0/3603546776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23900c6fd0 0x7f23901648b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:02:58.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 >> 172.21.15.32:0/3603546776 conn(0x7f23900cc7a0 msgr2=0x7f239006bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:02:58.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 shutdown_connections 2024-03-16T20:02:58.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:02:58.995+0000 7f2395ef1640 1 -- 172.21.15.32:0/3603546776 wait complete. 2024-03-16T20:02:59.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:02:59.598 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:00.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 Processor -- start 2024-03-16T20:03:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 -- start start 2024-03-16T20:03:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0f8072c70 0x7fe0f80c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0f806bae0 0x7fe0f8151330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f806e6a0 con 0x7fe0f8073050 2024-03-16T20:03:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f8156f20 con 0x7fe0f8072c70 2024-03-16T20:03:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0ff11c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f806c250 con 0x7fe0f806bae0 2024-03-16T20:03:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0f7fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe0f8073050 0x7fe0f8150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52282/0 (socket says 172.21.15.32:52282) 2024-03-16T20:03:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0f7fff640 1 -- 172.21.15.32:0/2057208745 learned_addr learned my addr 172.21.15.32:0/2057208745 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0fceb8640 1 --2- 172.21.15.32:0/2057208745 >> v2:172.21.15.32:3300/0 conn(0x7fe0f8072c70 0x7fe0f80c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.731+0000 7fe0fd6b9640 1 --2- 172.21.15.32:0/2057208745 >> v2:172.21.15.59:3300/0 conn(0x7fe0f806bae0 0x7fe0f8151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.735+0000 7fe0fd6b9640 1 -- 172.21.15.32:0/2057208745 >> v2:172.21.15.32:3300/0 conn(0x7fe0f8072c70 msgr2=0x7fe0f80c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.735+0000 7fe0fd6b9640 1 --2- 172.21.15.32:0/2057208745 >> v2:172.21.15.32:3300/0 conn(0x7fe0f8072c70 0x7fe0f80c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.735+0000 7fe0fd6b9640 1 -- 172.21.15.32:0/2057208745 >> v1:172.21.15.32:6789/0 conn(0x7fe0f8073050 legacy=0x7fe0f8150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.735+0000 7fe0fd6b9640 1 -- 172.21.15.32:0/2057208745 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f806c510 con 0x7fe0f806bae0 2024-03-16T20:03:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.739+0000 7fe0fd6b9640 1 --2- 172.21.15.32:0/2057208745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f8151330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe0f806f070 tx=0x7fe0f001c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07b06faf82537c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0f77fe640 1 -- 172.21.15.32:0/2057208745 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00a59e0 con 0x7fe0f806bae0 2024-03-16T20:03:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0f77fe640 1 -- 172.21.15.32:0/2057208745 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f00a5b40 con 0x7fe0f806bae0 2024-03-16T20:03:00.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0f77fe640 1 -- 172.21.15.32:0/2057208745 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00ad440 con 0x7fe0f806bae0 2024-03-16T20:03:00.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2057208745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 msgr2=0x7fe0f8151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:00.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2057208745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f8151330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe0f806f070 tx=0x7fe0f001c900 comp rx=0 tx=0).stop 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2057208745 shutdown_connections 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2057208745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f8151330 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2057208745 >> v2:172.21.15.32:3300/0 conn(0x7fe0f8072c70 0x7fe0f80c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2057208745 >> 172.21.15.32:0/2057208745 conn(0x7fe0f806afc0 msgr2=0x7fe0f806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2057208745 shutdown_connections 2024-03-16T20:03:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.747+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2057208745 wait complete. 2024-03-16T20:03:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 Processor -- start 2024-03-16T20:03:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 -- start start 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f81f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0f8073050 0x7fe0f80ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80cb690 con 0x7fe0f8072c70 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80ca0f0 con 0x7fe0f806bae0 2024-03-16T20:03:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0ff11c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81f7170 con 0x7fe0f8073050 2024-03-16T20:03:00.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0fd6b9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0f8073050 0x7fe0f80ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38182/0 (socket says 172.21.15.32:38182) 2024-03-16T20:03:00.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.751+0000 7fe0f7fff640 1 -- 172.21.15.32:0/2995877915 learned_addr learned my addr 172.21.15.32:0/2995877915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:00.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0fceb8640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f81f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0f7fff640 1 -- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0f8073050 msgr2=0x7fe0f80ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:00.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0f7fff640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0f8073050 0x7fe0f80ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:00.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0f7fff640 1 -- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 msgr2=0x7fe0f81f6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:00.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0f7fff640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f81f6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:00.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0f7fff640 1 -- 172.21.15.32:0/2995877915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0009a70 con 0x7fe0f8072c70 2024-03-16T20:03:00.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.755+0000 7fe0fceb8640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f81f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:03:00.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.759+0000 7fe0f7fff640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400c370 tx=0x7fe0e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.763+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4005cf0 con 0x7fe0f8072c70 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.763+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e4005e50 con 0x7fe0f8072c70 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.763+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f80c4120 con 0x7fe0f8072c70 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.767+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40072f0 con 0x7fe0f8072c70 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.767+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f80c43c0 con 0x7fe0f8072c70 2024-03-16T20:03:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.767+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f80674e0 con 0x7fe0f8072c70 2024-03-16T20:03:00.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.807+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe0e4007450 con 0x7fe0f8072c70 2024-03-16T20:03:00.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.807+0000 7fe0f57fa640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 0x7fe0dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:00.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.807+0000 7fe0fceb8640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 0x7fe0dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:00.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.807+0000 7fe0fceb8640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 0x7fe0dc069300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009bb0 tx=0x7fe0ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:00.811+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe0e4054ef0 con 0x7fe0f8072c70 2024-03-16T20:03:01.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.107+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe0e4051190 con 0x7fe0f8072c70 2024-03-16T20:03:01.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.307+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0f8068000 con 0x7fe0f8072c70 2024-03-16T20:03:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0f57fa640 1 -- 172.21.15.32:0/2995877915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe0e4017020 con 0x7fe0f8072c70 2024-03-16T20:03:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:01.345 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:02:02.472872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-03-16T20:03:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 msgr2=0x7fe0dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 0x7fe0dc069300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009bb0 tx=0x7fe0ec008040 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 msgr2=0x7fe0f806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400c370 tx=0x7fe0e400c830 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 shutdown_connections 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe0dc066e40 0x7fe0dc069300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0f8073050 0x7fe0f80ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8072c70 0x7fe0f806d3d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 --2- 172.21.15.32:0/2995877915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f806bae0 0x7fe0f81f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 >> 172.21.15.32:0/2995877915 conn(0x7fe0f806afc0 msgr2=0x7fe0f806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 shutdown_connections 2024-03-16T20:03:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:01.367+0000 7fe0ff11c640 1 -- 172.21.15.32:0/2995877915 wait complete. 2024-03-16T20:03:01.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:02:02.472872+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:01.974 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:02.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 Processor -- start 2024-03-16T20:03:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 -- start start 2024-03-16T20:03:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9afc154030 0x7f9afc072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc0c9de0 0x7f9afc159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9afc154410 0x7f9afc0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc06b990 con 0x7f9afc154410 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc06b3f0 con 0x7f9afc154030 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.163+0000 7f9b03fd9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc06b6b0 con 0x7f9afc0c9de0 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9afc154410 0x7f9afc0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9afc154410 0x7f9afc0cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38198/0 (socket says 172.21.15.32:38198) 2024-03-16T20:03:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 -- 172.21.15.32:0/497719670 learned_addr learned my addr 172.21.15.32:0/497719670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:03.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 -- 172.21.15.32:0/497719670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc0c9de0 msgr2=0x7f9afc159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 --2- 172.21.15.32:0/497719670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc0c9de0 0x7f9afc159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 -- 172.21.15.32:0/497719670 >> v2:172.21.15.59:3300/0 conn(0x7f9afc154030 msgr2=0x7f9afc072ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 --2- 172.21.15.32:0/497719670 >> v2:172.21.15.59:3300/0 conn(0x7f9afc154030 0x7f9afc072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.167+0000 7f9b02576640 1 -- 172.21.15.32:0/497719670 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0c9300 con 0x7f9afc154410 2024-03-16T20:03:03.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.171+0000 7f9b02576640 1 --2- 172.21.15.32:0/497719670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154410 0x7f9afc0cc620 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0ca200 tx=0x7f9af401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa0fb79853a8e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b00d73640 1 -- 172.21.15.32:0/497719670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40af400 con 0x7f9afc154410 2024-03-16T20:03:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b00d73640 1 -- 172.21.15.32:0/497719670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af40ab050 con 0x7f9afc154410 2024-03-16T20:03:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b00d73640 1 -- 172.21.15.32:0/497719670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40af780 con 0x7f9afc154410 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- 172.21.15.32:0/497719670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154410 msgr2=0x7f9afc0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/497719670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154410 0x7f9afc0cc620 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0ca200 tx=0x7f9af401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- 172.21.15.32:0/497719670 shutdown_connections 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/497719670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154410 0x7f9afc0cc620 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/497719670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc0c9de0 0x7f9afc159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/497719670 >> v2:172.21.15.59:3300/0 conn(0x7f9afc154030 0x7f9afc072ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- 172.21.15.32:0/497719670 >> 172.21.15.32:0/497719670 conn(0x7f9afc06e6c0 msgr2=0x7f9afc06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- 172.21.15.32:0/497719670 shutdown_connections 2024-03-16T20:03:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- 172.21.15.32:0/497719670 wait complete. 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 Processor -- start 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 -- start start 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9afc0c9de0 0x7f9afc0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.179+0000 7f9b03fd9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.183+0000 7f9b03fd9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc154410 0x7f9afc143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.183+0000 7f9b03fd9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc0ca4d0 con 0x7f9afc154030 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.183+0000 7f9b03fd9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc0cb940 con 0x7f9afc0c9de0 2024-03-16T20:03:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.183+0000 7f9b03fd9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc1efcc0 con 0x7f9afc154410 2024-03-16T20:03:03.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.183+0000 7f9b01574640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:03.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.187+0000 7f9b01574640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38214/0 (socket says 172.21.15.32:38214) 2024-03-16T20:03:03.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.187+0000 7f9b01574640 1 -- 172.21.15.32:0/599310689 learned_addr learned my addr 172.21.15.32:0/599310689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.187+0000 7f9b02576640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc154410 0x7f9afc143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.191+0000 7f9b01574640 1 -- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc154410 msgr2=0x7f9afc143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.191+0000 7f9b01574640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc154410 0x7f9afc143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.191+0000 7f9b01574640 1 -- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9afc0c9de0 msgr2=0x7f9afc0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.191+0000 7f9b01574640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9afc0c9de0 0x7f9afc0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.191+0000 7f9b01574640 1 -- 172.21.15.32:0/599310689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af4009b30 con 0x7f9afc154030 2024-03-16T20:03:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.195+0000 7f9b01574640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00b480 tx=0x7f9aec00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec010610 con 0x7f9afc154030 2024-03-16T20:03:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec010770 con 0x7f9afc154030 2024-03-16T20:03:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec010b20 con 0x7f9afc154030 2024-03-16T20:03:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc1eff60 con 0x7f9afc154030 2024-03-16T20:03:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc1f3bf0 con 0x7f9afc154030 2024-03-16T20:03:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.199+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc068000 con 0x7f9afc154030 2024-03-16T20:03:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.231+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9aec010c80 con 0x7f9afc154030 2024-03-16T20:03:03.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.231+0000 7f9af27fc640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 0x7f9ae0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:03.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.231+0000 7f9b01d75640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 0x7f9ae0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:03.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.235+0000 7f9b01d75640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 0x7f9ae0069350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009730 tx=0x7f9af8006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.235+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9aec055720 con 0x7f9afc154030 2024-03-16T20:03:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.523+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9aec0519c0 con 0x7f9afc154030 2024-03-16T20:03:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.799+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9afc0c8bb0 con 0x7f9afc154030 2024-03-16T20:03:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.835+0000 7f9af27fc640 1 -- 172.21.15.32:0/599310689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f9aec014090 con 0x7f9afc154030 2024-03-16T20:03:03.815 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:03.815 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 msgr2=0x7f9ae0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 0x7f9ae0069350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009730 tx=0x7f9af8006ca0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 msgr2=0x7f9afc1f03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00b480 tx=0x7f9aec00b940 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 shutdown_connections 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9ae0066e90 0x7f9ae0069350 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9afc154410 0x7f9afc143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9afc154030 0x7f9afc1f03f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 --2- 172.21.15.32:0/599310689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9afc0c9de0 0x7f9afc0700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 >> 172.21.15.32:0/599310689 conn(0x7f9afc06e6c0 msgr2=0x7f9afc1524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 shutdown_connections 2024-03-16T20:03:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:03.839+0000 7f9b03fd9640 1 -- 172.21.15.32:0/599310689 wait complete. 2024-03-16T20:03:04.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:04.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:05.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 Processor -- start 2024-03-16T20:03:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 -- start start 2024-03-16T20:03:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c150f90 0x7fe80c156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe80c0c87f0 0x7fe80c1616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c0700f0 con 0x7fe80c0c6fd0 2024-03-16T20:03:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c072e70 con 0x7fe80c0c87f0 2024-03-16T20:03:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe812012640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0728b0 con 0x7fe80c150f90 2024-03-16T20:03:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe80bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe80c0c87f0 0x7fe80c1616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe80affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe80c0c6fd0 0x7fe80c06c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52294/0 (socket says 172.21.15.32:52294) 2024-03-16T20:03:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe80affd640 1 -- 172.21.15.32:0/1502997354 learned_addr learned my addr 172.21.15.32:0/1502997354 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.567+0000 7fe80b7fe640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c150f90 0x7fe80c156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.571+0000 7fe80bfff640 1 -- 172.21.15.32:0/1502997354 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c150f90 msgr2=0x7fe80c156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.571+0000 7fe80bfff640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c150f90 0x7fe80c156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.571+0000 7fe80bfff640 1 -- 172.21.15.32:0/1502997354 >> v1:172.21.15.32:6789/0 conn(0x7fe80c0c6fd0 legacy=0x7fe80c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.575+0000 7fe80bfff640 1 -- 172.21.15.32:0/1502997354 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c072b90 con 0x7fe80c0c87f0 2024-03-16T20:03:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.579+0000 7fe80bfff640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c0c87f0 0x7fe80c1616b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe804009d90 tx=0x7fe80401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ecf72da2ff8b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.583+0000 7fe80a7fc640 1 -- 172.21.15.32:0/1502997354 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040a5940 con 0x7fe80c0c87f0 2024-03-16T20:03:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe80a7fc640 1 -- 172.21.15.32:0/1502997354 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8040a5aa0 con 0x7fe80c0c87f0 2024-03-16T20:03:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe80a7fc640 1 -- 172.21.15.32:0/1502997354 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040a5e30 con 0x7fe80c0c87f0 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- 172.21.15.32:0/1502997354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c0c87f0 msgr2=0x7fe80c1616b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c0c87f0 0x7fe80c1616b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe804009d90 tx=0x7fe80401c970 comp rx=0 tx=0).stop 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- 172.21.15.32:0/1502997354 shutdown_connections 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c0c87f0 0x7fe80c1616b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- 172.21.15.32:0/1502997354 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c150f90 0x7fe80c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- 172.21.15.32:0/1502997354 >> 172.21.15.32:0/1502997354 conn(0x7fe80c0cc7a0 msgr2=0x7fe80c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- 172.21.15.32:0/1502997354 shutdown_connections 2024-03-16T20:03:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- 172.21.15.32:0/1502997354 wait complete. 2024-03-16T20:03:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 Processor -- start 2024-03-16T20:03:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- start start 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c0c6fd0 0x7fe80c14b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0c87f0 0x7fe80c151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c161a70 con 0x7fe80c150f90 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c14d5b0 con 0x7fe80c0c87f0 2024-03-16T20:03:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.587+0000 7fe812012640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c14cdb0 con 0x7fe80c0c6fd0 2024-03-16T20:03:05.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.591+0000 7fe80bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.591+0000 7fe80affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0c87f0 0x7fe80c151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.591+0000 7fe80bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38234/0 (socket says 172.21.15.32:38234) 2024-03-16T20:03:05.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.591+0000 7fe80bfff640 1 -- 172.21.15.32:0/2468373127 learned_addr learned my addr 172.21.15.32:0/2468373127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:05.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.591+0000 7fe80b7fe640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c0c6fd0 0x7fe80c14b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 -- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c0c6fd0 msgr2=0x7fe80c14b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c0c6fd0 0x7fe80c14b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 -- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0c87f0 msgr2=0x7fe80c151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0c87f0 0x7fe80c151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 -- 172.21.15.32:0/2468373127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe804009a70 con 0x7fe80c150f90 2024-03-16T20:03:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.595+0000 7fe80bfff640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe8040b0040 tx=0x7fe804019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:05.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040addf0 con 0x7fe80c150f90 2024-03-16T20:03:05.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe804009450 con 0x7fe80c150f90 2024-03-16T20:03:05.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040b9690 con 0x7fe80c150f90 2024-03-16T20:03:05.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe812012640 1 -- 172.21.15.32:0/2468373127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c1614b0 con 0x7fe80c150f90 2024-03-16T20:03:05.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe812012640 1 -- 172.21.15.32:0/2468373127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c14b9e0 con 0x7fe80c150f90 2024-03-16T20:03:05.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.603+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e40052a0 con 0x7fe80c150f90 2024-03-16T20:03:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.623+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe8040b97f0 con 0x7fe80c150f90 2024-03-16T20:03:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.623+0000 7fe7f3fff640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 0x7fe7f8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.623+0000 7fe80b7fe640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 0x7fe7f8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.623+0000 7fe80b7fe640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 0x7fe7f8069380 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009ac0 tx=0x7fe7fc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.879+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe8040f0f80 con 0x7fe80c150f90 2024-03-16T20:03:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:05.879+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe8040ed220 con 0x7fe80c150f90 2024-03-16T20:03:06.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.075+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e4005900 con 0x7fe80c150f90 2024-03-16T20:03:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f3fff640 1 -- 172.21.15.32:0/2468373127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fe7e4005900 con 0x7fe80c150f90 2024-03-16T20:03:06.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:06.079 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 msgr2=0x7fe7f8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 0x7fe7f8069380 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009ac0 tx=0x7fe7fc009190 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 msgr2=0x7fe80c14c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe8040b0040 tx=0x7fe804019710 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 shutdown_connections 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe7f8066ec0 0x7fe7f8069380 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe80c150f90 0x7fe80c14c2c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0c87f0 0x7fe80c151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 --2- 172.21.15.32:0/2468373127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe80c0c6fd0 0x7fe80c14b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.099+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 >> 172.21.15.32:0/2468373127 conn(0x7fe80c0cc7a0 msgr2=0x7fe80c0c3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:06.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.103+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 shutdown_connections 2024-03-16T20:03:06.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:06.103+0000 7fe7f1ffb640 1 -- 172.21.15.32:0/2468373127 wait complete. 2024-03-16T20:03:06.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:06.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:07.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 Processor -- start 2024-03-16T20:03:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 -- start start 2024-03-16T20:03:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f077c151000 0x7f077c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077c0c7a40 con 0x7f077c156b30 2024-03-16T20:03:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077c0c9270 con 0x7f077c0c81b0 2024-03-16T20:03:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.831+0000 7f078407f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077c0c52e0 con 0x7f077c151000 2024-03-16T20:03:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.835+0000 7f078161a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f077c156b30 0x7f077c0c89e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52308/0 (socket says 172.21.15.32:52308) 2024-03-16T20:03:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.835+0000 7f078161a640 1 -- 172.21.15.32:0/3975617533 learned_addr learned my addr 172.21.15.32:0/3975617533 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.835+0000 7f0781e1b640 1 --2- 172.21.15.32:0/3975617533 >> v2:172.21.15.59:3300/0 conn(0x7f077c151000 0x7f077c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.835+0000 7f0780e19640 1 -- 172.21.15.32:0/3975617533 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434636720 0 0) 0x7f077c0c7a40 con 0x7f077c156b30 2024-03-16T20:03:07.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.835+0000 7f0780e19640 1 -- 172.21.15.32:0/3975617533 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0768003530 con 0x7f077c156b30 2024-03-16T20:03:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.843+0000 7f0781e1b640 1 -- 172.21.15.32:0/3975617533 >> v1:172.21.15.59:6789/0 conn(0x7f077c0c81b0 legacy=0x7f077c0c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.843+0000 7f0781e1b640 1 -- 172.21.15.32:0/3975617533 >> v1:172.21.15.32:6789/0 conn(0x7f077c156b30 legacy=0x7f077c0c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.843+0000 7f0781e1b640 1 -- 172.21.15.32:0/3975617533 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c0c9510 con 0x7f077c151000 2024-03-16T20:03:07.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.847+0000 7f0781e1b640 1 --2- 172.21.15.32:0/3975617533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c151000 0x7f077c0c70d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f07780097c0 tx=0x7f077801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07d9a5cca4ffeb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f0780e19640 1 -- 172.21.15.32:0/3975617533 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780a8990 con 0x7f077c151000 2024-03-16T20:03:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f0780e19640 1 -- 172.21.15.32:0/3975617533 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07780a8af0 con 0x7f077c151000 2024-03-16T20:03:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f0780e19640 1 -- 172.21.15.32:0/3975617533 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780a8e80 con 0x7f077c151000 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- 172.21.15.32:0/3975617533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c151000 msgr2=0x7f077c0c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 --2- 172.21.15.32:0/3975617533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c151000 0x7f077c0c70d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f07780097c0 tx=0x7f077801ec90 comp rx=0 tx=0).stop 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- 172.21.15.32:0/3975617533 shutdown_connections 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 --2- 172.21.15.32:0/3975617533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c151000 0x7f077c0c70d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- 172.21.15.32:0/3975617533 >> 172.21.15.32:0/3975617533 conn(0x7f077c06c0d0 msgr2=0x7f077c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- 172.21.15.32:0/3975617533 shutdown_connections 2024-03-16T20:03:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- 172.21.15.32:0/3975617533 wait complete. 2024-03-16T20:03:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 Processor -- start 2024-03-16T20:03:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.859+0000 7f078407f640 1 -- start start 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f077c0c81b0 0x7f077c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c156b30 0x7f077c1e4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1f2c40 con 0x7f077c151000 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1f2f20 con 0x7f077c156b30 2024-03-16T20:03:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078407f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077c155830 con 0x7f077c0c81b0 2024-03-16T20:03:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078161a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078161a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38242/0 (socket says 172.21.15.32:38242) 2024-03-16T20:03:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078161a640 1 -- 172.21.15.32:0/1984847966 learned_addr learned my addr 172.21.15.32:0/1984847966 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f0781e1b640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f077c0c81b0 0x7f077c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.863+0000 7f078261c640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c156b30 0x7f077c1e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 -- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f077c0c81b0 msgr2=0x7f077c0cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f077c0c81b0 0x7f077c0cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 -- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c156b30 msgr2=0x7f077c1e4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c156b30 0x7f077c1e4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 -- 172.21.15.32:0/1984847966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0778009420 con 0x7f077c151000 2024-03-16T20:03:07.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.867+0000 7f078161a640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c3f0 tx=0x7f076c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c005cf0 con 0x7f077c151000 2024-03-16T20:03:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c005e50 con 0x7f077c151000 2024-03-16T20:03:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c0cab60 con 0x7f077c151000 2024-03-16T20:03:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0072f0 con 0x7f077c151000 2024-03-16T20:03:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c0cb120 con 0x7f077c151000 2024-03-16T20:03:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:07.875+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c068000 con 0x7f077c151000 2024-03-16T20:03:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.171+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f076c007450 con 0x7f077c151000 2024-03-16T20:03:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.171+0000 7f07727fc640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 0x7f0760069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.171+0000 7f0781e1b640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 0x7f0760069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.171+0000 7f0781e1b640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 0x7f0760069330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07780b2000 tx=0x7f07780aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.207+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f076c055030 con 0x7f077c151000 2024-03-16T20:03:08.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.207+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f076c0512d0 con 0x7f077c151000 2024-03-16T20:03:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.419+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f077c1f3060 con 0x7f077c151000 2024-03-16T20:03:08.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f07727fc640 1 -- 172.21.15.32:0/1984847966 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f076c017070 con 0x7f077c151000 2024-03-16T20:03:08.421 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:08.421 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:08.421 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 msgr2=0x7f0760069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 0x7f0760069330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07780b2000 tx=0x7f07780aa070 comp rx=0 tx=0).stop 2024-03-16T20:03:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 msgr2=0x7f077c1f33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c3f0 tx=0x7f076c00c8b0 comp rx=0 tx=0).stop 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 shutdown_connections 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0760066e70 0x7f0760069330 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c156b30 0x7f077c1e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077c151000 0x7f077c1f33b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 --2- 172.21.15.32:0/1984847966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f077c0c81b0 0x7f077c0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 >> 172.21.15.32:0/1984847966 conn(0x7f077c06c0d0 msgr2=0x7f077c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 shutdown_connections 2024-03-16T20:03:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:08.443+0000 7f078407f640 1 -- 172.21.15.32:0/1984847966 wait complete. 2024-03-16T20:03:09.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:09.045 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:10.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 Processor -- start 2024-03-16T20:03:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 -- start start 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 0x7f55f80c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f55f80c97b0 0x7f55f80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f8158c80 con 0x7f55f80c8720 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f80b0ad0 con 0x7f55f80c97b0 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.175+0000 7f55fdb28640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f8157cb0 con 0x7f55f8151ef0 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.179+0000 7f55f7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f55f80c8720 0x7f55f815b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:41204/0 (socket says 172.21.15.32:41204) 2024-03-16T20:03:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.179+0000 7f55f7fff640 1 -- 172.21.15.32:0/2180755194 learned_addr learned my addr 172.21.15.32:0/2180755194 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.179+0000 7f55f77fe640 1 --2- 172.21.15.32:0/2180755194 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 0x7f55f80c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.179+0000 7f55f6ffd640 1 --2- 172.21.15.32:0/2180755194 >> v2:172.21.15.59:3300/0 conn(0x7f55f80c97b0 0x7f55f80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.187+0000 7f55f77fe640 1 -- 172.21.15.32:0/2180755194 >> v2:172.21.15.59:3300/0 conn(0x7f55f80c97b0 msgr2=0x7f55f80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.187+0000 7f55f77fe640 1 --2- 172.21.15.32:0/2180755194 >> v2:172.21.15.59:3300/0 conn(0x7f55f80c97b0 0x7f55f80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.187+0000 7f55f77fe640 1 -- 172.21.15.32:0/2180755194 >> v1:172.21.15.59:6789/0 conn(0x7f55f80c8720 legacy=0x7f55f815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.187+0000 7f55f77fe640 1 -- 172.21.15.32:0/2180755194 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f815bfc0 con 0x7f55f8151ef0 2024-03-16T20:03:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.191+0000 7f55f77fe640 1 --2- 172.21.15.32:0/2180755194 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 0x7f55f80c9270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f55e80097c0 tx=0x7f55e801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29b23c59c267ec70 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.199+0000 7f55f67fc640 1 -- 172.21.15.32:0/2180755194 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80a8990 con 0x7f55f8151ef0 2024-03-16T20:03:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.199+0000 7f55f67fc640 1 -- 172.21.15.32:0/2180755194 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e80a8af0 con 0x7f55f8151ef0 2024-03-16T20:03:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.199+0000 7f55f67fc640 1 -- 172.21.15.32:0/2180755194 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80a8e80 con 0x7f55f8151ef0 2024-03-16T20:03:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- 172.21.15.32:0/2180755194 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 msgr2=0x7f55f80c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- 172.21.15.32:0/2180755194 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 0x7f55f80c9270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f55e80097c0 tx=0x7f55e801ec90 comp rx=0 tx=0).stop 2024-03-16T20:03:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- 172.21.15.32:0/2180755194 shutdown_connections 2024-03-16T20:03:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- 172.21.15.32:0/2180755194 >> v2:172.21.15.59:3300/0 conn(0x7f55f80c97b0 0x7f55f80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- 172.21.15.32:0/2180755194 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f8151ef0 0x7f55f80c9270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- 172.21.15.32:0/2180755194 >> 172.21.15.32:0/2180755194 conn(0x7f55f8070f90 msgr2=0x7f55f80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- 172.21.15.32:0/2180755194 shutdown_connections 2024-03-16T20:03:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- 172.21.15.32:0/2180755194 wait complete. 2024-03-16T20:03:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 Processor -- start 2024-03-16T20:03:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- start start 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f80c8720 0x7f55f8150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f8151ef0 0x7f55f81f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80cd560 con 0x7f55f80c97b0 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c7840 con 0x7f55f8151ef0 2024-03-16T20:03:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.203+0000 7f55fdb28640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80ccbc0 con 0x7f55f80c8720 2024-03-16T20:03:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.207+0000 7f55f6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.207+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f8151ef0 0x7f55f81f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.207+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f80c8720 0x7f55f8150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.207+0000 7f55f6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45974/0 (socket says 172.21.15.32:45974) 2024-03-16T20:03:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.207+0000 7f55f6ffd640 1 -- 172.21.15.32:0/3033079454 learned_addr learned my addr 172.21.15.32:0/3033079454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.211+0000 7f55f6ffd640 1 -- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f80c8720 msgr2=0x7f55f8150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.211+0000 7f55f6ffd640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f80c8720 0x7f55f8150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.211+0000 7f55f6ffd640 1 -- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f8151ef0 msgr2=0x7f55f81f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.211+0000 7f55f6ffd640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f8151ef0 0x7f55f81f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.211+0000 7f55f6ffd640 1 -- 172.21.15.32:0/3033079454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e8009420 con 0x7f55f80c97b0 2024-03-16T20:03:10.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.215+0000 7f55f6ffd640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00d7d0 tx=0x7f55ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.219+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00b6c0 con 0x7f55f80c97b0 2024-03-16T20:03:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.219+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec00b820 con 0x7f55f80c97b0 2024-03-16T20:03:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.219+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00bbd0 con 0x7f55f80c97b0 2024-03-16T20:03:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.219+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f80ccea0 con 0x7f55f80c97b0 2024-03-16T20:03:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.219+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f80cc840 con 0x7f55f80c97b0 2024-03-16T20:03:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.223+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f8068000 con 0x7f55f80c97b0 2024-03-16T20:03:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.547+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f55ec00bd30 con 0x7f55f80c97b0 2024-03-16T20:03:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.551+0000 7f55dffff640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 0x7f55e4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.551+0000 7f55f77fe640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 0x7f55e4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:10.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.551+0000 7f55f77fe640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 0x7f55e4069330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f55e80b2000 tx=0x7f55e80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.567+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f55ec056540 con 0x7f55f80c97b0 2024-03-16T20:03:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.567+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f55ec0527e0 con 0x7f55f80c97b0 2024-03-16T20:03:10.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.795+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f8151570 con 0x7f55f80c97b0 2024-03-16T20:03:10.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55dffff640 1 -- 172.21.15.32:0/3033079454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f55ec055ea0 con 0x7f55f80c97b0 2024-03-16T20:03:10.903 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:10.903 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:10.903 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 msgr2=0x7f55e4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 0x7f55e4069330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f55e80b2000 tx=0x7f55e80aa070 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 msgr2=0x7f55f80b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00d7d0 tx=0x7f55ec00dc90 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 shutdown_connections 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f55e4066e70 0x7f55e4069330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f8151ef0 0x7f55f81f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f80c97b0 0x7f55f80b1f30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 --2- 172.21.15.32:0/3033079454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f55f80c8720 0x7f55f8150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 >> 172.21.15.32:0/3033079454 conn(0x7f55f8070f90 msgr2=0x7f55f8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 shutdown_connections 2024-03-16T20:03:10.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:10.923+0000 7f55fdb28640 1 -- 172.21.15.32:0/3033079454 wait complete. 2024-03-16T20:03:11.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:11.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:12.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:03:12.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:03:12.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 Processor -- start 2024-03-16T20:03:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 -- start start 2024-03-16T20:03:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3d8154350 0x7fc3d806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d80c2d70 0x7fc3d815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3d8072a10 0x7fc3d8072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d806bae0 con 0x7fc3d8154350 2024-03-16T20:03:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d815a540 con 0x7fc3d8072a10 2024-03-16T20:03:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dfd76640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d80c7fd0 con 0x7fc3d80c2d70 2024-03-16T20:03:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3ddb12640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3d8154350 0x7fc3d806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3dd311640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d80c2d70 0x7fc3d815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3ddb12640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3d8154350 0x7fc3d806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45990/0 (socket says 172.21.15.32:45990) 2024-03-16T20:03:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3ddb12640 1 -- 172.21.15.32:0/1718389416 learned_addr learned my addr 172.21.15.32:0/1718389416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.655+0000 7fc3de313640 1 --2- 172.21.15.32:0/1718389416 >> v2:172.21.15.59:3300/0 conn(0x7fc3d8072a10 0x7fc3d8072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.659+0000 7fc3ddb12640 1 -- 172.21.15.32:0/1718389416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d80c2d70 msgr2=0x7fc3d815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:12.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.659+0000 7fc3ddb12640 1 --2- 172.21.15.32:0/1718389416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d80c2d70 0x7fc3d815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.659+0000 7fc3ddb12640 1 -- 172.21.15.32:0/1718389416 >> v2:172.21.15.59:3300/0 conn(0x7fc3d8072a10 msgr2=0x7fc3d8072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:12.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.659+0000 7fc3ddb12640 1 --2- 172.21.15.32:0/1718389416 >> v2:172.21.15.59:3300/0 conn(0x7fc3d8072a10 0x7fc3d8072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.659+0000 7fc3ddb12640 1 -- 172.21.15.32:0/1718389416 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d806bd10 con 0x7fc3d8154350 2024-03-16T20:03:12.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.663+0000 7fc3ddb12640 1 --2- 172.21.15.32:0/1718389416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8154350 0x7fc3d806c250 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc3d806ca80 tx=0x7fc3d401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6076e889bb5f3f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:12.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dcb10640 1 -- 172.21.15.32:0/1718389416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40ab600 con 0x7fc3d8154350 2024-03-16T20:03:12.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dcb10640 1 -- 172.21.15.32:0/1718389416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40a7050 con 0x7fc3d8154350 2024-03-16T20:03:12.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dcb10640 1 -- 172.21.15.32:0/1718389416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40ab980 con 0x7fc3d8154350 2024-03-16T20:03:12.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1718389416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8154350 msgr2=0x7fc3d806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:12.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1718389416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8154350 0x7fc3d806c250 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc3d806ca80 tx=0x7fc3d401c470 comp rx=0 tx=0).stop 2024-03-16T20:03:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1718389416 shutdown_connections 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1718389416 >> v2:172.21.15.59:3300/0 conn(0x7fc3d8072a10 0x7fc3d8072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1718389416 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d80c2d70 0x7fc3d815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1718389416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8154350 0x7fc3d806c250 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1718389416 >> 172.21.15.32:0/1718389416 conn(0x7fc3d806afc0 msgr2=0x7fc3d806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1718389416 shutdown_connections 2024-03-16T20:03:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.667+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1718389416 wait complete. 2024-03-16T20:03:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 Processor -- start 2024-03-16T20:03:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 -- start start 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d8072a10 0x7fc3d81f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3d8154350 0x7fc3d8151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d81f38d0 con 0x7fc3d80c2d70 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8157440 con 0x7fc3d8154350 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dfd76640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8155af0 con 0x7fc3d8072a10 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dd311640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3ddb12640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d8072a10 0x7fc3d81f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dd311640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45992/0 (socket says 172.21.15.32:45992) 2024-03-16T20:03:12.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3dd311640 1 -- 172.21.15.32:0/1095959262 learned_addr learned my addr 172.21.15.32:0/1095959262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:12.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.671+0000 7fc3de313640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3d8154350 0x7fc3d8151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.675+0000 7fc3dd311640 1 -- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d8072a10 msgr2=0x7fc3d81f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:12.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.675+0000 7fc3dd311640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d8072a10 0x7fc3d81f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.675+0000 7fc3dd311640 1 -- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3d8154350 msgr2=0x7fc3d8151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:12.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.675+0000 7fc3dd311640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3d8154350 0x7fc3d8151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:12.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.675+0000 7fc3dd311640 1 -- 172.21.15.32:0/1095959262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40094d0 con 0x7fc3d80c2d70 2024-03-16T20:03:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.679+0000 7fc3dd311640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e930 tx=0x7fc3c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:12.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8009690 con 0x7fc3d80c2d70 2024-03-16T20:03:12.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c800ca00 con 0x7fc3d80c2d70 2024-03-16T20:03:12.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d8157700 con 0x7fc3d80c2d70 2024-03-16T20:03:12.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c800cd70 con 0x7fc3d80c2d70 2024-03-16T20:03:12.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d8154db0 con 0x7fc3d80c2d70 2024-03-16T20:03:12.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.683+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d80674e0 con 0x7fc3d80c2d70 2024-03-16T20:03:12.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.711+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc3c8013470 con 0x7fc3d80c2d70 2024-03-16T20:03:12.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.711+0000 7fc3ce7fc640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 0x7fc3bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:12.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.711+0000 7fc3ddb12640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 0x7fc3bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:12.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:12.711+0000 7fc3ddb12640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 0x7fc3bc069300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40ad000 tx=0x7fc3d40a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.031+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc3c80560f0 con 0x7fc3d80c2d70 2024-03-16T20:03:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.031+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc3c8052390 con 0x7fc3d80c2d70 2024-03-16T20:03:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.319+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d8068000 con 0x7fc3d80c2d70 2024-03-16T20:03:13.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3ce7fc640 1 -- 172.21.15.32:0/1095959262 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc3c8021020 con 0x7fc3d80c2d70 2024-03-16T20:03:13.370 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:13.370 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:13.370 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 msgr2=0x7fc3bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 0x7fc3bc069300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40ad000 tx=0x7fc3d40a6000 comp rx=0 tx=0).stop 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 msgr2=0x7fc3d81555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e930 tx=0x7fc3c800edf0 comp rx=0 tx=0).stop 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 shutdown_connections 2024-03-16T20:03:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc3bc066e40 0x7fc3bc069300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3d8154350 0x7fc3d8151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d80c2d70 0x7fc3d81555b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 --2- 172.21.15.32:0/1095959262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3d8072a10 0x7fc3d81f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 >> 172.21.15.32:0/1095959262 conn(0x7fc3d806afc0 msgr2=0x7fc3d8066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 shutdown_connections 2024-03-16T20:03:13.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:13.391+0000 7fc3dfd76640 1 -- 172.21.15.32:0/1095959262 wait complete. 2024-03-16T20:03:14.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:14.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:15.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.127+0000 7f7a095fa640 1 Processor -- start 2024-03-16T20:03:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.127+0000 7f7a095fa640 1 -- start start 2024-03-16T20:03:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a095fa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a040cdb80 0x7f7a0406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a095fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7a0406f8b0 0x7f7a04151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a095fa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a04157eb0 con 0x7f7a040cdf60 2024-03-16T20:03:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a095fa640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a040c4c80 con 0x7f7a040cdb80 2024-03-16T20:03:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a095fa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a040c4f40 con 0x7f7a0406f8b0 2024-03-16T20:03:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a037fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a040cdf60 0x7f7a04158340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47722/0 (socket says 172.21.15.32:47722) 2024-03-16T20:03:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a037fe640 1 -- 172.21.15.32:0/3451176635 learned_addr learned my addr 172.21.15.32:0/3451176635 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:15.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.131+0000 7f7a027fc640 1 --2- 172.21.15.32:0/3451176635 >> v2:172.21.15.59:3300/0 conn(0x7f7a0406f8b0 0x7f7a04151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.135+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3451176635 >> v2:172.21.15.32:3300/0 conn(0x7f7a040cdb80 0x7f7a0406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.135+0000 7f7a01ffb640 1 -- 172.21.15.32:0/3451176635 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083128887 0 0) 0x7f7a04157eb0 con 0x7f7a040cdf60 2024-03-16T20:03:15.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.135+0000 7f7a01ffb640 1 -- 172.21.15.32:0/3451176635 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ec003530 con 0x7f7a040cdf60 2024-03-16T20:03:15.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.139+0000 7f7a02ffd640 1 -- 172.21.15.32:0/3451176635 >> v2:172.21.15.59:3300/0 conn(0x7f7a0406f8b0 msgr2=0x7f7a04151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:15.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.139+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3451176635 >> v2:172.21.15.59:3300/0 conn(0x7f7a0406f8b0 0x7f7a04151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.139+0000 7f7a02ffd640 1 -- 172.21.15.32:0/3451176635 >> v1:172.21.15.32:6789/0 conn(0x7f7a040cdf60 legacy=0x7f7a04158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.139+0000 7f7a02ffd640 1 -- 172.21.15.32:0/3451176635 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040c5220 con 0x7f7a040cdb80 2024-03-16T20:03:15.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.143+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3451176635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdb80 0x7f7a0406f370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7a0406f300 tx=0x7f79f401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd343fc8edde97f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:15.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a01ffb640 1 -- 172.21.15.32:0/3451176635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40ab670 con 0x7f7a040cdb80 2024-03-16T20:03:15.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a01ffb640 1 -- 172.21.15.32:0/3451176635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f40ab7d0 con 0x7f7a040cdb80 2024-03-16T20:03:15.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a01ffb640 1 -- 172.21.15.32:0/3451176635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40abb60 con 0x7f7a040cdb80 2024-03-16T20:03:15.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 -- 172.21.15.32:0/3451176635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdb80 msgr2=0x7f7a0406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3451176635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdb80 0x7f7a0406f370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7a0406f300 tx=0x7f79f401c450 comp rx=0 tx=0).stop 2024-03-16T20:03:15.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 -- 172.21.15.32:0/3451176635 shutdown_connections 2024-03-16T20:03:15.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3451176635 >> v2:172.21.15.59:3300/0 conn(0x7f7a0406f8b0 0x7f7a04151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3451176635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdb80 0x7f7a0406f370 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 -- 172.21.15.32:0/3451176635 >> 172.21.15.32:0/3451176635 conn(0x7f7a04066ea0 msgr2=0x7f7a040672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:15.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 -- 172.21.15.32:0/3451176635 shutdown_connections 2024-03-16T20:03:15.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 -- 172.21.15.32:0/3451176635 wait complete. 2024-03-16T20:03:15.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.151+0000 7f7a095fa640 1 Processor -- start 2024-03-16T20:03:15.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 -- start start 2024-03-16T20:03:15.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a040cdb80 0x7f7a040c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a04148460 0x7f7a0414a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a041f0e50 con 0x7f7a040cdf60 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04159c20 con 0x7f7a040cdb80 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a095fa640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04148b70 con 0x7f7a04148460 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a027fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a027fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46004/0 (socket says 172.21.15.32:46004) 2024-03-16T20:03:15.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a027fc640 1 -- 172.21.15.32:0/3562391304 learned_addr learned my addr 172.21.15.32:0/3562391304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:15.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a040cdb80 0x7f7a040c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.155+0000 7f7a037fe640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a04148460 0x7f7a0414a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.159+0000 7f7a027fc640 1 -- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a04148460 msgr2=0x7f7a0414a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.159+0000 7f7a027fc640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a04148460 0x7f7a0414a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.159+0000 7f7a027fc640 1 -- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a040cdb80 msgr2=0x7f7a040c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.159+0000 7f7a027fc640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a040cdb80 0x7f7a040c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.159+0000 7f7a027fc640 1 -- 172.21.15.32:0/3562391304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4009420 con 0x7f7a040cdf60 2024-03-16T20:03:15.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.163+0000 7f7a027fc640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e900 tx=0x7f79f800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:15.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.163+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8009690 con 0x7f7a040cdf60 2024-03-16T20:03:15.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.163+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a041f08b0 con 0x7f7a040cdf60 2024-03-16T20:03:15.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.163+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a04148db0 con 0x7f7a040cdf60 2024-03-16T20:03:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.167+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a04068000 con 0x7f7a040cdf60 2024-03-16T20:03:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.374+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f80059a0 con 0x7f7a040cdf60 2024-03-16T20:03:15.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.418+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8005d10 con 0x7f7a040cdf60 2024-03-16T20:03:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.442+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f79f800c9a0 con 0x7f7a040cdf60 2024-03-16T20:03:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.442+0000 7f79eb7fe640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 0x7f79f0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.442+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 0x7f79f0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:15.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.446+0000 7f7a02ffd640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 0x7f79f0069300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79f40af000 tx=0x7f79f40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:15.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.462+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f79f8055f40 con 0x7f7a040cdf60 2024-03-16T20:03:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.462+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f79f80521e0 con 0x7f7a040cdf60 2024-03-16T20:03:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.662+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a04158620 con 0x7f7a040cdf60 2024-03-16T20:03:15.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.682+0000 7f79eb7fe640 1 -- 172.21.15.32:0/3562391304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f79f8021070 con 0x7f7a040cdf60 2024-03-16T20:03:15.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:15.665 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:02.995201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:03:02.995196+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:03:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 msgr2=0x7f79f0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 0x7f79f0069300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79f40af000 tx=0x7f79f40a7000 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 msgr2=0x7f7a0414a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e900 tx=0x7f79f800edc0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 shutdown_connections 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f79f0066e40 0x7f79f0069300 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a04148460 0x7f7a0414a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040cdf60 0x7f7a0414a460 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 --2- 172.21.15.32:0/3562391304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a040cdb80 0x7f7a040c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 >> 172.21.15.32:0/3562391304 conn(0x7f7a04066ea0 msgr2=0x7f7a041575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 shutdown_connections 2024-03-16T20:03:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:15.686+0000 7f7a095fa640 1 -- 172.21.15.32:0/3562391304 wait complete. 2024-03-16T20:03:16.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:02.995201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:03:02.995196+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:03:16.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:17.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:17.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.422+0000 7efcd937a640 1 Processor -- start 2024-03-16T20:03:17.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.422+0000 7efcd937a640 1 -- start start 2024-03-16T20:03:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd937a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd40cd280 0x7efcd40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd937a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcd40c8460 0x7efcd415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd937a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd415a9b0 con 0x7efcd41544a0 2024-03-16T20:03:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd937a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd415af50 con 0x7efcd40c8460 2024-03-16T20:03:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd937a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40c77c0 con 0x7efcd40cd280 2024-03-16T20:03:17.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efcd41544a0 0x7efcd40c7d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47736/0 (socket says 172.21.15.32:47736) 2024-03-16T20:03:17.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd2ffd640 1 -- 172.21.15.32:0/2102175478 learned_addr learned my addr 172.21.15.32:0/2102175478 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:17.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efcd37fe640 1 --2- 172.21.15.32:0/2102175478 >> v2:172.21.15.59:3300/0 conn(0x7efcd40c8460 0x7efcd415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.426+0000 7efccbfff640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd40cd280 0x7efcd40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd27fc640 1 -- 172.21.15.32:0/2102175478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584232418 0 0) 0x7efcd415a9b0 con 0x7efcd41544a0 2024-03-16T20:03:17.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd27fc640 1 -- 172.21.15.32:0/2102175478 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc0003530 con 0x7efcd41544a0 2024-03-16T20:03:17.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd37fe640 1 -- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd40cd280 msgr2=0x7efcd40730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:17.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd37fe640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd40cd280 0x7efcd40730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:17.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd37fe640 1 -- 172.21.15.32:0/2102175478 >> v1:172.21.15.32:6789/0 conn(0x7efcd41544a0 legacy=0x7efcd40c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:17.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.430+0000 7efcd37fe640 1 -- 172.21.15.32:0/2102175478 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd415ac70 con 0x7efcd40c8460 2024-03-16T20:03:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.438+0000 7efcd37fe640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40c8460 0x7efcd415d4a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fc897ffd5c558d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd27fc640 1 -- 172.21.15.32:0/2102175478 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0a5920 con 0x7efcd40c8460 2024-03-16T20:03:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd27fc640 1 -- 172.21.15.32:0/2102175478 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc0a5a80 con 0x7efcd40c8460 2024-03-16T20:03:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd27fc640 1 -- 172.21.15.32:0/2102175478 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0a5e10 con 0x7efcd40c8460 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 -- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40c8460 msgr2=0x7efcd415d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40c8460 0x7efcd415d4a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc01c970 comp rx=0 tx=0).stop 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 -- 172.21.15.32:0/2102175478 shutdown_connections 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40c8460 0x7efcd415d4a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 --2- 172.21.15.32:0/2102175478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd40cd280 0x7efcd40730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 -- 172.21.15.32:0/2102175478 >> 172.21.15.32:0/2102175478 conn(0x7efcd40cc5d0 msgr2=0x7efcd40cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 -- 172.21.15.32:0/2102175478 shutdown_connections 2024-03-16T20:03:17.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.446+0000 7efcd937a640 1 -- 172.21.15.32:0/2102175478 wait complete. 2024-03-16T20:03:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 Processor -- start 2024-03-16T20:03:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 -- start start 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 0x7efcd415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40cd280 0x7efcd4155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 0x7efcd4151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415ee70 con 0x7efcd40c8460 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd41600a0 con 0x7efcd40cd280 2024-03-16T20:03:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.450+0000 7efcd937a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415e8d0 con 0x7efcd41544a0 2024-03-16T20:03:17.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.454+0000 7efcd37fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 0x7efcd4151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.454+0000 7efccbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40cd280 0x7efcd4155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.454+0000 7efcd37fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 0x7efcd4151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:58596/0 (socket says 172.21.15.32:58596) 2024-03-16T20:03:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.454+0000 7efcd37fe640 1 -- 172.21.15.32:0/2213214153 learned_addr learned my addr 172.21.15.32:0/2213214153 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.454+0000 7efcd2ffd640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 0x7efcd415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 -- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 msgr2=0x7efcd4151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 0x7efcd4151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 -- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40cd280 msgr2=0x7efcd4155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40cd280 0x7efcd4155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 -- 172.21.15.32:0/2213214153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc009a70 con 0x7efcd40c8460 2024-03-16T20:03:17.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.458+0000 7efcd2ffd640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 0x7efcd415fb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efcc4001ec0 tx=0x7efcc400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4005cf0 con 0x7efcd40c8460 2024-03-16T20:03:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4005e50 con 0x7efcd40c8460 2024-03-16T20:03:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc40072d0 con 0x7efcd40c8460 2024-03-16T20:03:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd415eb90 con 0x7efcd40c8460 2024-03-16T20:03:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd41f6320 con 0x7efcd40c8460 2024-03-16T20:03:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.466+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd41f5ea0 con 0x7efcd40c8460 2024-03-16T20:03:17.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.498+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efcc4007430 con 0x7efcd40c8460 2024-03-16T20:03:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.498+0000 7efccb7fe640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 0x7efcb4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.498+0000 7efccbfff640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 0x7efcb4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:17.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.498+0000 7efccbfff640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 0x7efcb4069330 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efcd4151d70 tx=0x7efcbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.502+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efcc4054db0 con 0x7efcd40c8460 2024-03-16T20:03:17.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:17.814+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efcc4051050 con 0x7efcd40c8460 2024-03-16T20:03:18.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.026+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd4068000 con 0x7efcd40c8460 2024-03-16T20:03:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efccb7fe640 1 -- 172.21.15.32:0/2213214153 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7efcc4021090 con 0x7efcd40c8460 2024-03-16T20:03:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:18.076 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 msgr2=0x7efcb4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 0x7efcb4069330 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efcd4151d70 tx=0x7efcbc008040 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 msgr2=0x7efcd415fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 0x7efcd415fb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efcc4001ec0 tx=0x7efcc400c820 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 shutdown_connections 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcb4066e70 0x7efcb4069330 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcd41544a0 0x7efcd4151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcd40cd280 0x7efcd4155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 --2- 172.21.15.32:0/2213214153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd40c8460 0x7efcd415fb60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.094+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 >> 172.21.15.32:0/2213214153 conn(0x7efcd40cc5d0 msgr2=0x7efcd406f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.098+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 shutdown_connections 2024-03-16T20:03:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:18.098+0000 7efcd937a640 1 -- 172.21.15.32:0/2213214153 wait complete. 2024-03-16T20:03:18.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:18.703 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:19.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 Processor -- start 2024-03-16T20:03:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 -- start start 2024-03-16T20:03:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e74151220 0x7f9e740c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c4210 0x7f9e740c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740c3940 con 0x7f9e740c3e30 2024-03-16T20:03:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740c7ca0 con 0x7f9e74151220 2024-03-16T20:03:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e7b80b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74072cc0 con 0x7f9e740c4210 2024-03-16T20:03:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e78da6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e74151220 0x7f9e740c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e795a7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9e740c3e30 0x7f9e74150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:42352/0 (socket says 172.21.15.32:42352) 2024-03-16T20:03:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e795a7640 1 -- 172.21.15.32:0/2957379760 learned_addr learned my addr 172.21.15.32:0/2957379760 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.834+0000 7f9e79da8640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c4210 0x7f9e740c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.838+0000 7f9e78da6640 1 -- 172.21.15.32:0/2957379760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c4210 msgr2=0x7f9e740c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.838+0000 7f9e78da6640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c4210 0x7f9e740c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.838+0000 7f9e78da6640 1 -- 172.21.15.32:0/2957379760 >> v1:172.21.15.59:6789/0 conn(0x7f9e740c3e30 legacy=0x7f9e74150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.838+0000 7f9e78da6640 1 -- 172.21.15.32:0/2957379760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740c7780 con 0x7f9e74151220 2024-03-16T20:03:19.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.842+0000 7f9e78da6640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e740c80f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097c0 tx=0x7f9e6401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50514ffed39d8df server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.846+0000 7f9e6bfff640 1 -- 172.21.15.32:0/2957379760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640ad640 con 0x7f9e74151220 2024-03-16T20:03:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e6bfff640 1 -- 172.21.15.32:0/2957379760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e640ad7a0 con 0x7f9e74151220 2024-03-16T20:03:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e6bfff640 1 -- 172.21.15.32:0/2957379760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640adb30 con 0x7f9e74151220 2024-03-16T20:03:19.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- 172.21.15.32:0/2957379760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 msgr2=0x7f9e740c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e740c80f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097c0 tx=0x7f9e6401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- 172.21.15.32:0/2957379760 shutdown_connections 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c4210 0x7f9e740c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/2957379760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e740c80f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- 172.21.15.32:0/2957379760 >> 172.21.15.32:0/2957379760 conn(0x7f9e7406e6c0 msgr2=0x7f9e7406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- 172.21.15.32:0/2957379760 shutdown_connections 2024-03-16T20:03:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- 172.21.15.32:0/2957379760 wait complete. 2024-03-16T20:03:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 Processor -- start 2024-03-16T20:03:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- start start 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c3e30 0x7f9e741481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 0x7f9e740cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74148880 con 0x7f9e74151220 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7415a3f0 con 0x7f9e740c4210 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e7b80b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7415a810 con 0x7f9e740c3e30 2024-03-16T20:03:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e79da8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.850+0000 7f9e79da8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41590/0 (socket says 172.21.15.32:41590) 2024-03-16T20:03:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 -- 172.21.15.32:0/158463267 learned_addr learned my addr 172.21.15.32:0/158463267 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e795a7640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c3e30 0x7f9e741481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e78da6640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 0x7f9e740cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 -- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c3e30 msgr2=0x7f9e741481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c3e30 0x7f9e741481f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 -- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 msgr2=0x7f9e740cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 0x7f9e740cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e79da8640 1 -- 172.21.15.32:0/158463267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64009420 con 0x7f9e74151220 2024-03-16T20:03:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.854+0000 7f9e78da6640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 0x7f9e740cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:03:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.858+0000 7f9e79da8640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00efc0 tx=0x7f9e6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c009ca0 con 0x7f9e74151220 2024-03-16T20:03:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c00f040 con 0x7f9e74151220 2024-03-16T20:03:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740c8890 con 0x7f9e74151220 2024-03-16T20:03:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0146c0 con 0x7f9e74151220 2024-03-16T20:03:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e74159dc0 con 0x7f9e74151220 2024-03-16T20:03:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.862+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7415a530 con 0x7f9e74151220 2024-03-16T20:03:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.894+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9e6c014820 con 0x7f9e74151220 2024-03-16T20:03:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.894+0000 7f9e69ffb640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 0x7f9e58069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.894+0000 7f9e795a7640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 0x7f9e58069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.894+0000 7f9e795a7640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 0x7f9e58069350 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000f70 tx=0x7f9e70007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:19.898+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9e6c056f00 con 0x7f9e74151220 2024-03-16T20:03:20.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.166+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9e6c052aa0 con 0x7f9e74151220 2024-03-16T20:03:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.358+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7415a530 con 0x7f9e74151220 2024-03-16T20:03:20.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.382+0000 7f9e69ffb640 1 -- 172.21.15.32:0/158463267 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9e7415a530 con 0x7f9e74151220 2024-03-16T20:03:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:20.366 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:20.366 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 msgr2=0x7f9e58069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 0x7f9e58069350 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000f70 tx=0x7f9e70007e90 comp rx=0 tx=0).stop 2024-03-16T20:03:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 msgr2=0x7f9e741f0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00efc0 tx=0x7f9e6c00c490 comp rx=0 tx=0).stop 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 shutdown_connections 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e58066e90 0x7f9e58069350 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74151220 0x7f9e741f0c50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e740c4210 0x7f9e740cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 --2- 172.21.15.32:0/158463267 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e740c3e30 0x7f9e741481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 >> 172.21.15.32:0/158463267 conn(0x7f9e7406e6c0 msgr2=0x7f9e740c5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 shutdown_connections 2024-03-16T20:03:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:20.386+0000 7f9e7b80b640 1 -- 172.21.15.32:0/158463267 wait complete. 2024-03-16T20:03:20.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:20.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:21.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 Processor -- start 2024-03-16T20:03:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 -- start start 2024-03-16T20:03:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f871c06de40 0x7f871c15a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c15a390 con 0x7f871c06e220 2024-03-16T20:03:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c071b70 con 0x7f871c06de40 2024-03-16T20:03:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f872350b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c072bd0 con 0x7f871c0c73f0 2024-03-16T20:03:22.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f8720aa6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f87212a7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f871c06de40 0x7f871c15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f8720aa6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:51918/0 (socket says 172.21.15.32:51918) 2024-03-16T20:03:22.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.122+0000 7f8720aa6640 1 -- 172.21.15.32:0/1298784167 learned_addr learned my addr 172.21.15.32:0/1298784167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.130+0000 7f8720aa6640 1 -- 172.21.15.32:0/1298784167 >> v2:172.21.15.59:3300/0 conn(0x7f871c06de40 msgr2=0x7f871c15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.130+0000 7f8720aa6640 1 --2- 172.21.15.32:0/1298784167 >> v2:172.21.15.59:3300/0 conn(0x7f871c06de40 0x7f871c15a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.130+0000 7f8720aa6640 1 -- 172.21.15.32:0/1298784167 >> v1:172.21.15.59:6789/0 conn(0x7f871c06e220 legacy=0x7f871c15d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.130+0000 7f8720aa6640 1 -- 172.21.15.32:0/1298784167 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c072eb0 con 0x7f871c0c73f0 2024-03-16T20:03:22.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.134+0000 7f8720aa6640 1 --2- 172.21.15.32:0/1298784167 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f870c009870 tx=0x7f870c01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af80732576103a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:22.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f8713fff640 1 -- 172.21.15.32:0/1298784167 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0a89d0 con 0x7f871c0c73f0 2024-03-16T20:03:22.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f8713fff640 1 -- 172.21.15.32:0/1298784167 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c0a8b30 con 0x7f871c0c73f0 2024-03-16T20:03:22.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f8713fff640 1 -- 172.21.15.32:0/1298784167 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0b0440 con 0x7f871c0c73f0 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- 172.21.15.32:0/1298784167 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 msgr2=0x7f871c153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 --2- 172.21.15.32:0/1298784167 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f870c009870 tx=0x7f870c01ed10 comp rx=0 tx=0).stop 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- 172.21.15.32:0/1298784167 shutdown_connections 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 --2- 172.21.15.32:0/1298784167 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c0c73f0 0x7f871c153e50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 --2- 172.21.15.32:0/1298784167 >> v2:172.21.15.59:3300/0 conn(0x7f871c06de40 0x7f871c15a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- 172.21.15.32:0/1298784167 >> 172.21.15.32:0/1298784167 conn(0x7f871c0cbfe0 msgr2=0x7f871c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- 172.21.15.32:0/1298784167 shutdown_connections 2024-03-16T20:03:22.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- 172.21.15.32:0/1298784167 wait complete. 2024-03-16T20:03:22.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 Processor -- start 2024-03-16T20:03:22.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.146+0000 7f872350b640 1 -- start start 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c06de40 0x7f871c151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1501d0 0x7f871c14a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c14f4a0 con 0x7f871c06e220 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1f34f0 con 0x7f871c1501d0 2024-03-16T20:03:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f872350b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c14ac20 con 0x7f871c06de40 2024-03-16T20:03:22.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f8720aa6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f8721aa8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1501d0 0x7f871c14a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f8720aa6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41602/0 (socket says 172.21.15.32:41602) 2024-03-16T20:03:22.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f8720aa6640 1 -- 172.21.15.32:0/1689674133 learned_addr learned my addr 172.21.15.32:0/1689674133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:22.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.150+0000 7f87212a7640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c06de40 0x7f871c151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 -- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c06de40 msgr2=0x7f871c151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c06de40 0x7f871c151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 -- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1501d0 msgr2=0x7f871c14a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1501d0 0x7f871c14a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 -- 172.21.15.32:0/1689674133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c0094d0 con 0x7f871c06e220 2024-03-16T20:03:22.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.154+0000 7f8720aa6640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f870c0b3040 tx=0x7f870c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0b0dd0 con 0x7f871c06e220 2024-03-16T20:03:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c0bb460 con 0x7f871c06e220 2024-03-16T20:03:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0bb810 con 0x7f871c06e220 2024-03-16T20:03:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c153890 con 0x7f871c06e220 2024-03-16T20:03:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c14f6f0 con 0x7f871c06e220 2024-03-16T20:03:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.162+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c0674e0 con 0x7f871c06e220 2024-03-16T20:03:22.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.198+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f870c0b83f0 con 0x7f871c06e220 2024-03-16T20:03:22.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.202+0000 7f8711ffb640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 0x7f8700069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:22.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.202+0000 7f87212a7640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 0x7f8700069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:22.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.202+0000 7f87212a7640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 0x7f8700069380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8718009b60 tx=0x7f8718009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.478+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f870c0f3400 con 0x7f871c06e220 2024-03-16T20:03:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.478+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f870c0ef6a0 con 0x7f871c06e220 2024-03-16T20:03:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.698+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f871c068000 con 0x7f871c06e220 2024-03-16T20:03:22.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.738+0000 7f8711ffb640 1 -- 172.21.15.32:0/1689674133 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f870c0b5090 con 0x7f871c06e220 2024-03-16T20:03:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:22.722 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 msgr2=0x7f8700069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 0x7f8700069380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8718009b60 tx=0x7f8718009190 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 msgr2=0x7f871c14c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f870c0b3040 tx=0x7f870c01bab0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 shutdown_connections 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8700066ec0 0x7f8700069380 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1501d0 0x7f871c14a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871c06e220 0x7f871c14c180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 --2- 172.21.15.32:0/1689674133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f871c06de40 0x7f871c151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 >> 172.21.15.32:0/1689674133 conn(0x7f871c0cbfe0 msgr2=0x7f871c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 shutdown_connections 2024-03-16T20:03:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:22.742+0000 7f872350b640 1 -- 172.21.15.32:0/1689674133 wait complete. 2024-03-16T20:03:23.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:23.343 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:24.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:24.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 Processor -- start 2024-03-16T20:03:24.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 -- start start 2024-03-16T20:03:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6138070840 0x7f61380c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6138070c90 con 0x7f6138153fa0 2024-03-16T20:03:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380c8590 con 0x7f61380724a0 2024-03-16T20:03:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.474+0000 7f61400c0640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61380c3f00 con 0x7f6138070840 2024-03-16T20:03:24.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.478+0000 7f613e65d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f61380724a0 0x7f6138164770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:42358/0 (socket says 172.21.15.32:42358) 2024-03-16T20:03:24.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.478+0000 7f613e65d640 1 -- 172.21.15.32:0/1179864574 learned_addr learned my addr 172.21.15.32:0/1179864574 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.486+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674720653 0 0) 0x7f61380c8590 con 0x7f61380724a0 2024-03-16T20:03:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.486+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6124003530 con 0x7f61380724a0 2024-03-16T20:03:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.498+0000 7f613d65b640 1 --2- 172.21.15.32:0/1179864574 >> v2:172.21.15.32:3300/0 conn(0x7f6138070840 0x7f61380c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359592033 0 0) 0x7f6124003530 con 0x7f61380724a0 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 >> v2:172.21.15.32:3300/0 conn(0x7f6138070840 msgr2=0x7f61380c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 --2- 172.21.15.32:0/1179864574 >> v2:172.21.15.32:3300/0 conn(0x7f6138070840 0x7f61380c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 >> v1:172.21.15.32:6789/0 conn(0x7f6138153fa0 legacy=0x7f613815d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138073590 con 0x7f61380724a0 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f6130001a00 con 0x7f61380724a0 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 >> v1:172.21.15.59:6789/0 conn(0x7f61380724a0 legacy=0x7f6138164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 0x7f6124003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f61240047d0 0x7f6124004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240051e0 0x7f612400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612400c0f0 con 0x7f61240039d0 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612400c260 con 0x7f61240051e0 2024-03-16T20:03:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.502+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612400c420 con 0x7f61240047d0 2024-03-16T20:03:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.506+0000 7f613e65d640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240051e0 0x7f612400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.506+0000 7f613de5c640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 0x7f6124003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.506+0000 7f613d65b640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f61240047d0 0x7f6124004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.510+0000 7f613de5c640 1 -- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f61240047d0 msgr2=0x7f6124004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.510+0000 7f613de5c640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f61240047d0 0x7f6124004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.510+0000 7f613de5c640 1 -- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240051e0 msgr2=0x7f612400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.510+0000 7f613de5c640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240051e0 0x7f612400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.510+0000 7f613de5c640 1 -- 172.21.15.32:0/1179864574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124003530 con 0x7f61240039d0 2024-03-16T20:03:24.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.514+0000 7f613e65d640 1 -- 172.21.15.32:0/1179864574 reap_dead start 2024-03-16T20:03:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.514+0000 7f613de5c640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 0x7f6124003df0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f61340015e0 tx=0x7f613401c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9450ff9852374f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.518+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61340052d0 con 0x7f61240039d0 2024-03-16T20:03:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.518+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61340a7050 con 0x7f61240039d0 2024-03-16T20:03:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.518+0000 7f613ce5a640 1 -- 172.21.15.32:0/1179864574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134003840 con 0x7f61240039d0 2024-03-16T20:03:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 msgr2=0x7f6124003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 0x7f6124003df0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f61340015e0 tx=0x7f613401c280 comp rx=0 tx=0).stop 2024-03-16T20:03:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- 172.21.15.32:0/1179864574 shutdown_connections 2024-03-16T20:03:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 --2- 172.21.15.32:0/1179864574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61240039d0 0x7f6124003df0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- 172.21.15.32:0/1179864574 >> 172.21.15.32:0/1179864574 conn(0x7f613806c0d0 msgr2=0x7f613806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- 172.21.15.32:0/1179864574 shutdown_connections 2024-03-16T20:03:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- 172.21.15.32:0/1179864574 wait complete. 2024-03-16T20:03:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 Processor -- start 2024-03-16T20:03:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- start start 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240039d0 0x7f6138142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6138147650 0x7f6138147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381f3cc0 con 0x7f61381f3170 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6138148010 con 0x7f61240039d0 2024-03-16T20:03:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.522+0000 7f61400c0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6138143710 con 0x7f6138147650 2024-03-16T20:03:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.526+0000 7f613d65b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.526+0000 7f613e65d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6138147650 0x7f6138147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.526+0000 7f613d65b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41626/0 (socket says 172.21.15.32:41626) 2024-03-16T20:03:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.526+0000 7f613d65b640 1 -- 172.21.15.32:0/2828149334 learned_addr learned my addr 172.21.15.32:0/2828149334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.526+0000 7f613de5c640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240039d0 0x7f6138142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 -- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6138147650 msgr2=0x7f6138147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6138147650 0x7f6138147ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 -- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240039d0 msgr2=0x7f6138142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240039d0 0x7f6138142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 -- 172.21.15.32:0/2828149334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138142e70 con 0x7f61381f3170 2024-03-16T20:03:24.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.530+0000 7f613d65b640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f612800b7f0 tx=0x7f612800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128010830 con 0x7f61381f3170 2024-03-16T20:03:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6128010990 con 0x7f61381f3170 2024-03-16T20:03:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128010d40 con 0x7f61381f3170 2024-03-16T20:03:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6138143150 con 0x7f61381f3170 2024-03-16T20:03:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6138146e50 con 0x7f61381f3170 2024-03-16T20:03:24.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.538+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6138068000 con 0x7f61381f3170 2024-03-16T20:03:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.562+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6128027480 con 0x7f61381f3170 2024-03-16T20:03:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.562+0000 7f612e7fc640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 0x7f611c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.562+0000 7f613de5c640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 0x7f611c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.562+0000 7f613de5c640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 0x7f611c069300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f61340015b0 tx=0x7f61340011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.870+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f61280561e0 con 0x7f61381f3170 2024-03-16T20:03:24.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:24.870+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6128052480 con 0x7f61381f3170 2024-03-16T20:03:25.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.262+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6138149ee0 con 0x7f61381f3170 2024-03-16T20:03:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f612e7fc640 1 -- 172.21.15.32:0/2828149334 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6128022090 con 0x7f61381f3170 2024-03-16T20:03:25.270 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:25.270 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 msgr2=0x7f611c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 0x7f611c069300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f61340015b0 tx=0x7f61340011a0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 msgr2=0x7f61381495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f612800b7f0 tx=0x7f612800bcb0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 shutdown_connections 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f611c066e40 0x7f611c069300 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6138147650 0x7f6138147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381f3170 0x7f61381495f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 --2- 172.21.15.32:0/2828149334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61240039d0 0x7f6138142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 >> 172.21.15.32:0/2828149334 conn(0x7f613806c0d0 msgr2=0x7f61380c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 shutdown_connections 2024-03-16T20:03:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:25.290+0000 7f61400c0640 1 -- 172.21.15.32:0/2828149334 wait complete. 2024-03-16T20:03:25.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:25.894 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:26.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:27.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 Processor -- start 2024-03-16T20:03:27.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 -- start start 2024-03-16T20:03:27.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a44151000 0x7f5a440c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440c8ec0 con 0x7f5a440c36b0 2024-03-16T20:03:27.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a44072f10 con 0x7f5a44151000 2024-03-16T20:03:27.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.022+0000 7f5a4a966640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440c5000 con 0x7f5a440c8980 2024-03-16T20:03:27.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a44151000 0x7f5a440c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a48f03640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a44151000 0x7f5a440c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41642/0 (socket says 172.21.15.32:41642) 2024-03-16T20:03:27.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 -- 172.21.15.32:0/2361203006 learned_addr learned my addr 172.21.15.32:0/2361203006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:27.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 -- 172.21.15.32:0/2361203006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 msgr2=0x7f5a4415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:27.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 --2- 172.21.15.32:0/2361203006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 -- 172.21.15.32:0/2361203006 >> v1:172.21.15.59:6789/0 conn(0x7f5a440c36b0 legacy=0x7f5a440c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.026+0000 7f5a43fff640 1 -- 172.21.15.32:0/2361203006 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44072950 con 0x7f5a44151000 2024-03-16T20:03:27.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.030+0000 7f5a43fff640 1 --2- 172.21.15.32:0/2361203006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a440c8420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009870 tx=0x7f5a3401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39780ded0bfd0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:27.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a42ffd640 1 -- 172.21.15.32:0/2361203006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340a9620 con 0x7f5a44151000 2024-03-16T20:03:27.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a42ffd640 1 -- 172.21.15.32:0/2361203006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a340a9780 con 0x7f5a44151000 2024-03-16T20:03:27.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a42ffd640 1 -- 172.21.15.32:0/2361203006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340a9b10 con 0x7f5a44151000 2024-03-16T20:03:27.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- 172.21.15.32:0/2361203006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 msgr2=0x7f5a440c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 --2- 172.21.15.32:0/2361203006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a440c8420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009870 tx=0x7f5a3401c560 comp rx=0 tx=0).stop 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- 172.21.15.32:0/2361203006 shutdown_connections 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 --2- 172.21.15.32:0/2361203006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 --2- 172.21.15.32:0/2361203006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a440c8420 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- 172.21.15.32:0/2361203006 >> 172.21.15.32:0/2361203006 conn(0x7f5a4406c0d0 msgr2=0x7f5a4406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:27.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- 172.21.15.32:0/2361203006 shutdown_connections 2024-03-16T20:03:27.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- 172.21.15.32:0/2361203006 wait complete. 2024-03-16T20:03:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 Processor -- start 2024-03-16T20:03:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.038+0000 7f5a4a966640 1 -- start start 2024-03-16T20:03:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a440c36b0 0x7f5a44143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4413a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4413ad40 con 0x7f5a44151000 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4414f240 con 0x7f5a440c36b0 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a4a966640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44142e40 con 0x7f5a440c8980 2024-03-16T20:03:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a48f03640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a437fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4413a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a440c36b0 0x7f5a44143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a48f03640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41648/0 (socket says 172.21.15.32:41648) 2024-03-16T20:03:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.042+0000 7f5a48f03640 1 -- 172.21.15.32:0/1155424518 learned_addr learned my addr 172.21.15.32:0/1155424518 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:27.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.046+0000 7f5a48f03640 1 -- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 msgr2=0x7f5a4413a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.046+0000 7f5a48f03640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4413a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.046+0000 7f5a48f03640 1 -- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a440c36b0 msgr2=0x7f5a44143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.046+0000 7f5a48f03640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a440c36b0 0x7f5a44143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.046+0000 7f5a48f03640 1 -- 172.21.15.32:0/1155424518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340094d0 con 0x7f5a44151000 2024-03-16T20:03:27.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.050+0000 7f5a48f03640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00efc0 tx=0x7f5a3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.054+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c014480 con 0x7f5a44151000 2024-03-16T20:03:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.054+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00f040 con 0x7f5a44151000 2024-03-16T20:03:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.054+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c014820 con 0x7f5a44151000 2024-03-16T20:03:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.054+0000 7f5a4a966640 1 -- 172.21.15.32:0/1155424518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44140470 con 0x7f5a44151000 2024-03-16T20:03:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.054+0000 7f5a4a966640 1 -- 172.21.15.32:0/1155424518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4414f490 con 0x7f5a44151000 2024-03-16T20:03:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.058+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c0052a0 con 0x7f5a44151000 2024-03-16T20:03:27.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.078+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5a3c014980 con 0x7f5a44151000 2024-03-16T20:03:27.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.078+0000 7f5a40ff9640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 0x7f5a30069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:27.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.078+0000 7f5a43fff640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 0x7f5a30069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:27.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.078+0000 7f5a43fff640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 0x7f5a30069350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5a340ae000 tx=0x7f5a340b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:27.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.358+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f5a3c056970 con 0x7f5a44151000 2024-03-16T20:03:27.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.358+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5a3c052c10 con 0x7f5a44151000 2024-03-16T20:03:27.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.562+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a1c005900 con 0x7f5a44151000 2024-03-16T20:03:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a40ff9640 1 -- 172.21.15.32:0/1155424518 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f5a1c005900 con 0x7f5a44151000 2024-03-16T20:03:27.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:27.574 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 msgr2=0x7f5a30069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 0x7f5a30069350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5a340ae000 tx=0x7f5a340b3040 comp rx=0 tx=0).stop 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 msgr2=0x7f5a4413a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00efc0 tx=0x7f5a3c00c490 comp rx=0 tx=0).stop 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 shutdown_connections 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5a30066e90 0x7f5a30069350 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a44151000 0x7f5a4413a800 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a440c8980 0x7f5a4413a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 --2- 172.21.15.32:0/1155424518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a440c36b0 0x7f5a44143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 >> 172.21.15.32:0/1155424518 conn(0x7f5a4406c0d0 msgr2=0x7f5a4406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 shutdown_connections 2024-03-16T20:03:27.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:27.594+0000 7f5a2a7fc640 1 -- 172.21.15.32:0/1155424518 wait complete. 2024-03-16T20:03:28.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:28.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:29.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 Processor -- start 2024-03-16T20:03:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 -- start start 2024-03-16T20:03:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef640c8b90 0x7fef6415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6415ad50 con 0x7fef64153fc0 2024-03-16T20:03:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef640c7ce0 con 0x7fef640c8b90 2024-03-16T20:03:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6c6d1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6415a1f0 con 0x7fef6415df30 2024-03-16T20:03:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef6ac6e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.330+0000 7fef69c6c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef640c8b90 0x7fef6415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.334+0000 7fef6ac6e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:47530/0 (socket says 172.21.15.32:47530) 2024-03-16T20:03:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.334+0000 7fef6ac6e640 1 -- 172.21.15.32:0/1347674410 learned_addr learned my addr 172.21.15.32:0/1347674410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.338+0000 7fef6ac6e640 1 -- 172.21.15.32:0/1347674410 >> v2:172.21.15.59:3300/0 conn(0x7fef640c8b90 msgr2=0x7fef6415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.338+0000 7fef6ac6e640 1 --2- 172.21.15.32:0/1347674410 >> v2:172.21.15.59:3300/0 conn(0x7fef640c8b90 0x7fef6415d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.338+0000 7fef6ac6e640 1 -- 172.21.15.32:0/1347674410 >> v1:172.21.15.59:6789/0 conn(0x7fef64153fc0 legacy=0x7fef640c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.338+0000 7fef6ac6e640 1 -- 172.21.15.32:0/1347674410 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6415a4b0 con 0x7fef6415df30 2024-03-16T20:03:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.342+0000 7fef6ac6e640 1 --2- 172.21.15.32:0/1347674410 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009e50 tx=0x7fef5c01f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b88eee827d6083b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6946b640 1 -- 172.21.15.32:0/1347674410 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0af400 con 0x7fef6415df30 2024-03-16T20:03:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6946b640 1 -- 172.21.15.32:0/1347674410 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c0ab050 con 0x7fef6415df30 2024-03-16T20:03:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6946b640 1 -- 172.21.15.32:0/1347674410 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0af780 con 0x7fef6415df30 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- 172.21.15.32:0/1347674410 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 msgr2=0x7fef640c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/1347674410 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009e50 tx=0x7fef5c01f1e0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- 172.21.15.32:0/1347674410 shutdown_connections 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/1347674410 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef640c8050 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/1347674410 >> v2:172.21.15.59:3300/0 conn(0x7fef640c8b90 0x7fef6415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- 172.21.15.32:0/1347674410 >> 172.21.15.32:0/1347674410 conn(0x7fef640cc7a0 msgr2=0x7fef640ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- 172.21.15.32:0/1347674410 shutdown_connections 2024-03-16T20:03:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- 172.21.15.32:0/1347674410 wait complete. 2024-03-16T20:03:29.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 Processor -- start 2024-03-16T20:03:29.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.350+0000 7fef6c6d1640 1 -- start start 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef64153fc0 0x7fef6414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef6415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6415b6d0 con 0x7fef640c8b90 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef641e6e40 con 0x7fef64153fc0 2024-03-16T20:03:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6c6d1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef641f06f0 con 0x7fef6415df30 2024-03-16T20:03:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6a46d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6a46d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50692/0 (socket says 172.21.15.32:50692) 2024-03-16T20:03:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6a46d640 1 -- 172.21.15.32:0/437177860 learned_addr learned my addr 172.21.15.32:0/437177860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef69c6c640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef64153fc0 0x7fef6414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.354+0000 7fef6ac6e640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef6415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.358+0000 7fef6a46d640 1 -- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 msgr2=0x7fef6415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.358+0000 7fef6a46d640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef6415cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.358+0000 7fef6a46d640 1 -- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef64153fc0 msgr2=0x7fef6414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.358+0000 7fef6a46d640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef64153fc0 0x7fef6414ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.358+0000 7fef6a46d640 1 -- 172.21.15.32:0/437177860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c009b30 con 0x7fef640c8b90 2024-03-16T20:03:29.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.362+0000 7fef6a46d640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c450 tx=0x7fef6000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.366+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60012830 con 0x7fef640c8b90 2024-03-16T20:03:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.366+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef600129b0 con 0x7fef640c8b90 2024-03-16T20:03:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.366+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60012d60 con 0x7fef640c8b90 2024-03-16T20:03:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.366+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef641f09d0 con 0x7fef640c8b90 2024-03-16T20:03:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.366+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef641e6ad0 con 0x7fef640c8b90 2024-03-16T20:03:29.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.370+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef64068000 con 0x7fef640c8b90 2024-03-16T20:03:29.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.406+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef6001c460 con 0x7fef640c8b90 2024-03-16T20:03:29.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.406+0000 7fef5affd640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 0x7fef48069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:29.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.406+0000 7fef69c6c640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 0x7fef48069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:29.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.406+0000 7fef69c6c640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 0x7fef48069380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef640c3640 tx=0x7fef54009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:29.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.410+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fef60055de0 con 0x7fef640c8b90 2024-03-16T20:03:29.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.710+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fef60052080 con 0x7fef640c8b90 2024-03-16T20:03:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:29.946+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef64157a90 con 0x7fef640c8b90 2024-03-16T20:03:30.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.054+0000 7fef5affd640 1 -- 172.21.15.32:0/437177860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fef60055740 con 0x7fef640c8b90 2024-03-16T20:03:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:30.035 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:30.035 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 msgr2=0x7fef48069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 0x7fef48069380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef640c3640 tx=0x7fef54009190 comp rx=0 tx=0).stop 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 msgr2=0x7fef64151390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c450 tx=0x7fef6000c910 comp rx=0 tx=0).stop 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 shutdown_connections 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef48066ec0 0x7fef48069380 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:30.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef6415df30 0x7fef6415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef64153fc0 0x7fef6414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 --2- 172.21.15.32:0/437177860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef640c8b90 0x7fef64151390 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 >> 172.21.15.32:0/437177860 conn(0x7fef640cc7a0 msgr2=0x7fef640ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 shutdown_connections 2024-03-16T20:03:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:30.058+0000 7fef6c6d1640 1 -- 172.21.15.32:0/437177860 wait complete. 2024-03-16T20:03:30.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:30.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:31.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:31.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 Processor -- start 2024-03-16T20:03:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 -- start start 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9c80725e0 0x7fe9c8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9c815a420 0x7fe9c815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8072dd0 con 0x7fe9c8150d70 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c815d790 con 0x7fe9c815a420 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.786+0000 7fe9cd580640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c815da50 con 0x7fe9c80725e0 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c6ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe9c8150d70 0x7fe9c8070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46260/0 (socket says 172.21.15.32:46260) 2024-03-16T20:03:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c6ffd640 1 -- 172.21.15.32:0/4110589624 learned_addr learned my addr 172.21.15.32:0/4110589624 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c67fc640 1 --2- 172.21.15.32:0/4110589624 >> v2:172.21.15.59:3300/0 conn(0x7fe9c80725e0 0x7fe9c8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c77fe640 1 --2- 172.21.15.32:0/4110589624 >> v2:172.21.15.32:3300/0 conn(0x7fe9c815a420 0x7fe9c815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c5ffb640 1 -- 172.21.15.32:0/4110589624 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692951596 0 0) 0x7fe9c8072dd0 con 0x7fe9c8150d70 2024-03-16T20:03:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.790+0000 7fe9c5ffb640 1 -- 172.21.15.32:0/4110589624 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003530 con 0x7fe9c8150d70 2024-03-16T20:03:31.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.794+0000 7fe9c67fc640 1 -- 172.21.15.32:0/4110589624 >> v2:172.21.15.32:3300/0 conn(0x7fe9c815a420 msgr2=0x7fe9c815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:31.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.794+0000 7fe9c67fc640 1 --2- 172.21.15.32:0/4110589624 >> v2:172.21.15.32:3300/0 conn(0x7fe9c815a420 0x7fe9c815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:31.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.794+0000 7fe9c67fc640 1 -- 172.21.15.32:0/4110589624 >> v1:172.21.15.32:6789/0 conn(0x7fe9c8150d70 legacy=0x7fe9c8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:31.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.794+0000 7fe9c67fc640 1 -- 172.21.15.32:0/4110589624 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c815dd30 con 0x7fe9c80725e0 2024-03-16T20:03:31.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.802+0000 7fe9c67fc640 1 --2- 172.21.15.32:0/4110589624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c8073070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9c806ef10 tx=0x7fe9bc01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a86cc4d2d82b564c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:31.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9c5ffb640 1 -- 172.21.15.32:0/4110589624 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0ab600 con 0x7fe9c80725e0 2024-03-16T20:03:31.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9c5ffb640 1 -- 172.21.15.32:0/4110589624 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0a7050 con 0x7fe9c80725e0 2024-03-16T20:03:31.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9c5ffb640 1 -- 172.21.15.32:0/4110589624 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0ab980 con 0x7fe9c80725e0 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 -- 172.21.15.32:0/4110589624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 msgr2=0x7fe9c8073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 --2- 172.21.15.32:0/4110589624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c8073070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9c806ef10 tx=0x7fe9bc01c330 comp rx=0 tx=0).stop 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 -- 172.21.15.32:0/4110589624 shutdown_connections 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 --2- 172.21.15.32:0/4110589624 >> v2:172.21.15.32:3300/0 conn(0x7fe9c815a420 0x7fe9c815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 --2- 172.21.15.32:0/4110589624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c8073070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 -- 172.21.15.32:0/4110589624 >> 172.21.15.32:0/4110589624 conn(0x7fe9c806c0d0 msgr2=0x7fe9c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 -- 172.21.15.32:0/4110589624 shutdown_connections 2024-03-16T20:03:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.810+0000 7fe9cd580640 1 -- 172.21.15.32:0/4110589624 wait complete. 2024-03-16T20:03:31.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 Processor -- start 2024-03-16T20:03:31.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 -- start start 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c81f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c8150d70 0x7fe9c814fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c814f6e0 con 0x7fe9c815a420 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c814e370 con 0x7fe9c80725e0 2024-03-16T20:03:31.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9cd580640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c81f6ac0 con 0x7fe9c8150d70 2024-03-16T20:03:31.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9c77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9c67fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c8150d70 0x7fe9c814fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9c77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50706/0 (socket says 172.21.15.32:50706) 2024-03-16T20:03:31.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.814+0000 7fe9c77fe640 1 -- 172.21.15.32:0/890307712 learned_addr learned my addr 172.21.15.32:0/890307712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:31.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c6ffd640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c81f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c77fe640 1 -- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c8150d70 msgr2=0x7fe9c814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c77fe640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c8150d70 0x7fe9c814fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c77fe640 1 -- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 msgr2=0x7fe9c81f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c77fe640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c81f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.818+0000 7fe9c77fe640 1 -- 172.21.15.32:0/890307712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc009420 con 0x7fe9c815a420 2024-03-16T20:03:31.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.822+0000 7fe9c77fe640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0014480 con 0x7fe9c815a420 2024-03-16T20:03:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c000f040 con 0x7fe9c815a420 2024-03-16T20:03:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c815d560 con 0x7fe9c815a420 2024-03-16T20:03:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0014820 con 0x7fe9c815a420 2024-03-16T20:03:31.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c814e910 con 0x7fe9c815a420 2024-03-16T20:03:31.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.826+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c814f8a0 con 0x7fe9c815a420 2024-03-16T20:03:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.850+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9c0014980 con 0x7fe9c815a420 2024-03-16T20:03:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.850+0000 7fe9af7fe640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 0x7fe9b4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.850+0000 7fe9c6ffd640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 0x7fe9b4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:31.850+0000 7fe9c6ffd640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 0x7fe9b4069350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8007fd0 tx=0x7fe9b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.130+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe9c0056840 con 0x7fe9c815a420 2024-03-16T20:03:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.130+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe9c0052ae0 con 0x7fe9c815a420 2024-03-16T20:03:32.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.326+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c814eb00 con 0x7fe9c815a420 2024-03-16T20:03:32.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9af7fe640 1 -- 172.21.15.32:0/890307712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe9c0018030 con 0x7fe9c815a420 2024-03-16T20:03:32.331 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:32.331 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:32.332 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 msgr2=0x7fe9b4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 0x7fe9b4069350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8007fd0 tx=0x7fe9b8009040 comp rx=0 tx=0).stop 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 msgr2=0x7fe9c81500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c490 comp rx=0 tx=0).stop 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 shutdown_connections 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe9b4066e90 0x7fe9b4069350 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c815a420 0x7fe9c81500e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9c8150d70 0x7fe9c814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 --2- 172.21.15.32:0/890307712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9c80725e0 0x7fe9c81f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 >> 172.21.15.32:0/890307712 conn(0x7fe9c806c0d0 msgr2=0x7fe9c80ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 shutdown_connections 2024-03-16T20:03:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:32.350+0000 7fe9cd580640 1 -- 172.21.15.32:0/890307712 wait complete. 2024-03-16T20:03:32.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:32.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:33.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.082+0000 7f499379f640 1 Processor -- start 2024-03-16T20:03:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.082+0000 7f499379f640 1 -- start start 2024-03-16T20:03:34.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f498c151000 0x7f498c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f498c0c36b0 0x7f498c15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c0c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c072f10 con 0x7f498c0c36b0 2024-03-16T20:03:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c0c5000 con 0x7f498c151000 2024-03-16T20:03:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499379f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0c8ec0 con 0x7f498c0c8980 2024-03-16T20:03:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f4990d3a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f498c0c36b0 0x7f498c15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f499153b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f498c151000 0x7f498c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f4990d3a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f498c0c36b0 0x7f498c15a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50708/0 (socket says 172.21.15.32:50708) 2024-03-16T20:03:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f4990d3a640 1 -- 172.21.15.32:0/142258256 learned_addr learned my addr 172.21.15.32:0/142258256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.086+0000 7f4991d3c640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c0c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4990d3a640 1 -- 172.21.15.32:0/142258256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 msgr2=0x7f498c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4990d3a640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c0c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4990d3a640 1 -- 172.21.15.32:0/142258256 >> v2:172.21.15.59:3300/0 conn(0x7f498c151000 msgr2=0x7f498c0c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4990d3a640 1 --2- 172.21.15.32:0/142258256 >> v2:172.21.15.59:3300/0 conn(0x7f498c151000 0x7f498c0c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4990d3a640 1 -- 172.21.15.32:0/142258256 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c072950 con 0x7f498c0c36b0 2024-03-16T20:03:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.090+0000 7f4991d3c640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c0c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:03:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.094+0000 7f4990d3a640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c15a240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f498c0725d0 tx=0x7f497c01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81fa4e42a8a05bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f4983fff640 1 -- 172.21.15.32:0/142258256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c0a7c80 con 0x7f498c0c36b0 2024-03-16T20:03:34.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f4983fff640 1 -- 172.21.15.32:0/142258256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c0a7de0 con 0x7f498c0c36b0 2024-03-16T20:03:34.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f4983fff640 1 -- 172.21.15.32:0/142258256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c0af660 con 0x7f498c0c36b0 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 -- 172.21.15.32:0/142258256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 msgr2=0x7f498c15a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c15a240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f498c0725d0 tx=0x7f497c01ebf0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 -- 172.21.15.32:0/142258256 shutdown_connections 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c0c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 --2- 172.21.15.32:0/142258256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c15a240 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 --2- 172.21.15.32:0/142258256 >> v2:172.21.15.59:3300/0 conn(0x7f498c151000 0x7f498c0c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 -- 172.21.15.32:0/142258256 >> 172.21.15.32:0/142258256 conn(0x7f498c06c0d0 msgr2=0x7f498c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 -- 172.21.15.32:0/142258256 shutdown_connections 2024-03-16T20:03:34.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.098+0000 7f499379f640 1 -- 172.21.15.32:0/142258256 wait complete. 2024-03-16T20:03:34.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 Processor -- start 2024-03-16T20:03:34.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 -- start start 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f498c151000 0x7f498c157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14bd90 con 0x7f498c0c36b0 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0cad30 con 0x7f498c151000 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499379f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14bfe0 con 0x7f498c0c8980 2024-03-16T20:03:34.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499153b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f4991d3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f498c151000 0x7f498c157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499153b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50724/0 (socket says 172.21.15.32:50724) 2024-03-16T20:03:34.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.102+0000 7f499153b640 1 -- 172.21.15.32:0/103347606 learned_addr learned my addr 172.21.15.32:0/103347606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:34.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.106+0000 7f4990d3a640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 -- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 msgr2=0x7f498c155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 -- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f498c151000 msgr2=0x7f498c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f498c151000 0x7f498c157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 -- 172.21.15.32:0/103347606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c00b770 con 0x7f498c0c36b0 2024-03-16T20:03:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.110+0000 7f499153b640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f498800e870 tx=0x7f498800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988009510 con 0x7f498c0c36b0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800eec0 con 0x7f498c0c36b0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c0ca770 con 0x7f498c0c36b0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800cb40 con 0x7f498c0c36b0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c1571f0 con 0x7f498c0c36b0 2024-03-16T20:03:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.118+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c068000 con 0x7f498c0c36b0 2024-03-16T20:03:34.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.410+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f498800cca0 con 0x7f498c0c36b0 2024-03-16T20:03:34.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.410+0000 7f4981ffb640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 0x7f4970069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:34.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.410+0000 7f4990d3a640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 0x7f4970069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:34.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.414+0000 7f4990d3a640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 0x7f4970069350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f497c0b1000 tx=0x7f497c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.434+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4988055e80 con 0x7f498c0c36b0 2024-03-16T20:03:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.434+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4988052120 con 0x7f498c0c36b0 2024-03-16T20:03:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.662+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f498c073410 con 0x7f498c0c36b0 2024-03-16T20:03:34.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.686+0000 7f4981ffb640 1 -- 172.21.15.32:0/103347606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4988017090 con 0x7f498c0c36b0 2024-03-16T20:03:34.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:34.670 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:34.670 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 msgr2=0x7f4970069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 0x7f4970069350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f497c0b1000 tx=0x7f497c008040 comp rx=0 tx=0).stop 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 msgr2=0x7f498c0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f498800e870 tx=0x7f498800ec70 comp rx=0 tx=0).stop 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 shutdown_connections 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4970066e90 0x7f4970069350 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f498c151000 0x7f498c157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f498c0c8980 0x7f498c155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 --2- 172.21.15.32:0/103347606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f498c0c36b0 0x7f498c0cc7e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:34.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 >> 172.21.15.32:0/103347606 conn(0x7f498c06c0d0 msgr2=0x7f498c0c6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:34.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 shutdown_connections 2024-03-16T20:03:34.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:34.690+0000 7f499379f640 1 -- 172.21.15.32:0/103347606 wait complete. 2024-03-16T20:03:35.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:35.301 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:36.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 Processor -- start 2024-03-16T20:03:36.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 -- start start 2024-03-16T20:03:36.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f28040c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f280415b2a0 0x7f280415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2804158c80 con 0x7f2804151ef0 2024-03-16T20:03:36.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28040b0ad0 con 0x7f280415b2a0 2024-03-16T20:03:36.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f280af03640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2804157cb0 con 0x7f28040c81e0 2024-03-16T20:03:36.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f2808c9f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2804151ef0 0x7f28040c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46274/0 (socket says 172.21.15.32:46274) 2024-03-16T20:03:36.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f2808c9f640 1 -- 172.21.15.32:0/2164653978 learned_addr learned my addr 172.21.15.32:0/2164653978 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:36.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.434+0000 7f2803fff640 1 --2- 172.21.15.32:0/2164653978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f28040c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.438+0000 7f28094a0640 1 --2- 172.21.15.32:0/2164653978 >> v2:172.21.15.32:3300/0 conn(0x7f280415b2a0 0x7f280415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.442+0000 7f2803fff640 1 -- 172.21.15.32:0/2164653978 >> v2:172.21.15.32:3300/0 conn(0x7f280415b2a0 msgr2=0x7f280415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:36.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.442+0000 7f2803fff640 1 --2- 172.21.15.32:0/2164653978 >> v2:172.21.15.32:3300/0 conn(0x7f280415b2a0 0x7f280415b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:36.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.442+0000 7f2803fff640 1 -- 172.21.15.32:0/2164653978 >> v1:172.21.15.32:6789/0 conn(0x7f2804151ef0 legacy=0x7f28040c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:36.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.442+0000 7f2803fff640 1 -- 172.21.15.32:0/2164653978 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280415bfc0 con 0x7f28040c81e0 2024-03-16T20:03:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.446+0000 7f2803fff640 1 --2- 172.21.15.32:0/2164653978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f28040c8600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097c0 tx=0x7f27f001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62662473b62f5110 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:36.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.450+0000 7f28037fe640 1 -- 172.21.15.32:0/2164653978 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00a8990 con 0x7f28040c81e0 2024-03-16T20:03:36.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.450+0000 7f28037fe640 1 -- 172.21.15.32:0/2164653978 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f00a8af0 con 0x7f28040c81e0 2024-03-16T20:03:36.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.450+0000 7f28037fe640 1 -- 172.21.15.32:0/2164653978 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00a8e80 con 0x7f28040c81e0 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- 172.21.15.32:0/2164653978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 msgr2=0x7f28040c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- 172.21.15.32:0/2164653978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f28040c8600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097c0 tx=0x7f27f001ec90 comp rx=0 tx=0).stop 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- 172.21.15.32:0/2164653978 shutdown_connections 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- 172.21.15.32:0/2164653978 >> v2:172.21.15.32:3300/0 conn(0x7f280415b2a0 0x7f280415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- 172.21.15.32:0/2164653978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f28040c8600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- 172.21.15.32:0/2164653978 >> 172.21.15.32:0/2164653978 conn(0x7f2804070f90 msgr2=0x7f28040713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- 172.21.15.32:0/2164653978 shutdown_connections 2024-03-16T20:03:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- 172.21.15.32:0/2164653978 wait complete. 2024-03-16T20:03:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 Processor -- start 2024-03-16T20:03:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- start start 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f280414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f280415b2a0 0x7f280414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28041f36e0 con 0x7f2804151ef0 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280406d780 con 0x7f280415b2a0 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f280af03640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280414c4b0 con 0x7f28040c81e0 2024-03-16T20:03:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f2803fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f2803fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50742/0 (socket says 172.21.15.32:50742) 2024-03-16T20:03:36.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f2803fff640 1 -- 172.21.15.32:0/2096458842 learned_addr learned my addr 172.21.15.32:0/2096458842 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:36.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.454+0000 7f28094a0640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f280415b2a0 0x7f280414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2808c9f640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f280414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2803fff640 1 -- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 msgr2=0x7f280414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:36.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2803fff640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f280414bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:36.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2803fff640 1 -- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f280415b2a0 msgr2=0x7f280414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:36.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2803fff640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f280415b2a0 0x7f280414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:36.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.458+0000 7f2803fff640 1 -- 172.21.15.32:0/2096458842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0009420 con 0x7f2804151ef0 2024-03-16T20:03:36.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.462+0000 7f2803fff640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f27f00b3040 tx=0x7f27f001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:36.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.466+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00b9630 con 0x7f2804151ef0 2024-03-16T20:03:36.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.466+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f00b97b0 con 0x7f2804151ef0 2024-03-16T20:03:36.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.466+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280406da20 con 0x7f2804151ef0 2024-03-16T20:03:36.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.466+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00b9b60 con 0x7f2804151ef0 2024-03-16T20:03:36.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.466+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280406dfe0 con 0x7f2804151ef0 2024-03-16T20:03:36.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.470+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28041f3820 con 0x7f2804151ef0 2024-03-16T20:03:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.750+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f27f00b7040 con 0x7f2804151ef0 2024-03-16T20:03:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.750+0000 7f28017fa640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 0x7f27e8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.750+0000 7f2808c9f640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 0x7f27e8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.754+0000 7f2808c9f640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 0x7f27e8069330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f70 tx=0x7f27f8007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.778+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f27f00f4000 con 0x7f2804151ef0 2024-03-16T20:03:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:36.778+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f27f00f02a0 con 0x7f2804151ef0 2024-03-16T20:03:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.034+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28041f3820 con 0x7f2804151ef0 2024-03-16T20:03:37.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.118+0000 7f28017fa640 1 -- 172.21.15.32:0/2096458842 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f28041f3820 con 0x7f2804151ef0 2024-03-16T20:03:37.100 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:37.100 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:37.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 msgr2=0x7f27e8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:37.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 0x7f27e8069330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f70 tx=0x7f27f8007e90 comp rx=0 tx=0).stop 2024-03-16T20:03:37.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 msgr2=0x7f28040cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:37.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f27f00b3040 tx=0x7f27f001ba30 comp rx=0 tx=0).stop 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 shutdown_connections 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f27e8066e70 0x7f27e8069330 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f280415b2a0 0x7f280414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2804151ef0 0x7f28040cc890 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 --2- 172.21.15.32:0/2096458842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28040c81e0 0x7f280414bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 >> 172.21.15.32:0/2096458842 conn(0x7f2804070f90 msgr2=0x7f280415a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 shutdown_connections 2024-03-16T20:03:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:37.122+0000 7f280af03640 1 -- 172.21.15.32:0/2096458842 wait complete. 2024-03-16T20:03:37.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:37.730 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:38.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.858+0000 7f6f07fc6640 1 Processor -- start 2024-03-16T20:03:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.858+0000 7f6f07fc6640 1 -- start start 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f07fc6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f00158980 0x7f6f00163740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f07fc6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000c68e0 con 0x7f6f000b1eb0 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f07fc6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f00067340 con 0x7f6f000c7ce0 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f07fc6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0006c1f0 con 0x7f6f00158980 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f05d62640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6f000b1eb0 0x7f6f00072a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46290/0 (socket says 172.21.15.32:46290) 2024-03-16T20:03:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f05d62640 1 -- 172.21.15.32:0/369964780 learned_addr learned my addr 172.21.15.32:0/369964780 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:38.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.862+0000 7f6f06563640 1 --2- 172.21.15.32:0/369964780 >> v2:172.21.15.59:3300/0 conn(0x7f6f00158980 0x7f6f00163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.866+0000 7f6f04d60640 1 -- 172.21.15.32:0/369964780 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236396746 0 0) 0x7f6f000c68e0 con 0x7f6f000b1eb0 2024-03-16T20:03:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.866+0000 7f6f04d60640 1 -- 172.21.15.32:0/369964780 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec003530 con 0x7f6f000b1eb0 2024-03-16T20:03:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.870+0000 7f6f06563640 1 -- 172.21.15.32:0/369964780 >> v1:172.21.15.59:6789/0 conn(0x7f6f000c7ce0 legacy=0x7f6f00158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.870+0000 7f6f06563640 1 -- 172.21.15.32:0/369964780 >> v1:172.21.15.32:6789/0 conn(0x7f6f000b1eb0 legacy=0x7f6f00072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.870+0000 7f6f06563640 1 -- 172.21.15.32:0/369964780 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0006c4d0 con 0x7f6f00158980 2024-03-16T20:03:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.878+0000 7f6f06563640 1 --2- 172.21.15.32:0/369964780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00158980 0x7f6f00163740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009d90 tx=0x7f6ef801f140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca318a89cc39c822 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f04d60640 1 -- 172.21.15.32:0/369964780 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80af600 con 0x7f6f00158980 2024-03-16T20:03:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f04d60640 1 -- 172.21.15.32:0/369964780 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef80ab050 con 0x7f6f00158980 2024-03-16T20:03:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f04d60640 1 -- 172.21.15.32:0/369964780 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80af980 con 0x7f6f00158980 2024-03-16T20:03:38.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 -- 172.21.15.32:0/369964780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00158980 msgr2=0x7f6f00163740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/369964780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00158980 0x7f6f00163740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009d90 tx=0x7f6ef801f140 comp rx=0 tx=0).stop 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 -- 172.21.15.32:0/369964780 shutdown_connections 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/369964780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f00158980 0x7f6f00163740 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 -- 172.21.15.32:0/369964780 >> 172.21.15.32:0/369964780 conn(0x7f6f0006afc0 msgr2=0x7f6f0006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 -- 172.21.15.32:0/369964780 shutdown_connections 2024-03-16T20:03:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.886+0000 7f6f07fc6640 1 -- 172.21.15.32:0/369964780 wait complete. 2024-03-16T20:03:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 Processor -- start 2024-03-16T20:03:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 -- start start 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f000c7ce0 0x7f6f000cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00158980 0x7f6f000c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001f3b00 con 0x7f6f000b1eb0 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001594e0 con 0x7f6f000c7ce0 2024-03-16T20:03:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f07fc6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0014dda0 con 0x7f6f00158980 2024-03-16T20:03:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f05d62640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f05d62640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50744/0 (socket says 172.21.15.32:50744) 2024-03-16T20:03:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.890+0000 7f6f05d62640 1 -- 172.21.15.32:0/2655710057 learned_addr learned my addr 172.21.15.32:0/2655710057 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05561640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f000c7ce0 0x7f6f000cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f06563640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00158980 0x7f6f000c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05d62640 1 -- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00158980 msgr2=0x7f6f000c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05d62640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00158980 0x7f6f000c1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05d62640 1 -- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f000c7ce0 msgr2=0x7f6f000cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05d62640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f000c7ce0 0x7f6f000cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.894+0000 7f6f05d62640 1 -- 172.21.15.32:0/2655710057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8009a70 con 0x7f6f000b1eb0 2024-03-16T20:03:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.898+0000 7f6f05d62640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c450 tx=0x7f6efc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.902+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc005cf0 con 0x7f6f000b1eb0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.902+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc005e50 con 0x7f6f000b1eb0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.902+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f001f2ec0 con 0x7f6f000b1eb0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.902+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc007290 con 0x7f6f000b1eb0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.902+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f001f3160 con 0x7f6f000b1eb0 2024-03-16T20:03:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:38.906+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f001f3c40 con 0x7f6f000b1eb0 2024-03-16T20:03:39.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.226+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6efc0073f0 con 0x7f6f000b1eb0 2024-03-16T20:03:39.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.226+0000 7f6ef67fc640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 0x7f6ee4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.226+0000 7f6f05561640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 0x7f6ee4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.226+0000 7f6f05561640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 0x7f6ee4069350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f0014d370 tx=0x7f6ef0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:39.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.242+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6efc054e50 con 0x7f6f000b1eb0 2024-03-16T20:03:39.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.242+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6efc0510f0 con 0x7f6f000b1eb0 2024-03-16T20:03:39.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.466+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f001f3c40 con 0x7f6f000b1eb0 2024-03-16T20:03:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6ef67fc640 1 -- 172.21.15.32:0/2655710057 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6f001f3c40 con 0x7f6f000b1eb0 2024-03-16T20:03:39.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:39.534 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 msgr2=0x7f6ee4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 0x7f6ee4069350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f0014d370 tx=0x7f6ef0009040 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 msgr2=0x7f6f00156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c450 tx=0x7f6efc00c910 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 shutdown_connections 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6ee4066e90 0x7f6ee4069350 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f00158980 0x7f6f000c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f000c7ce0 0x7f6f000cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 --2- 172.21.15.32:0/2655710057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f000b1eb0 0x7f6f00156100 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 >> 172.21.15.32:0/2655710057 conn(0x7f6f0006afc0 msgr2=0x7f6f00071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:39.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 shutdown_connections 2024-03-16T20:03:39.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:39.554+0000 7f6f07fc6640 1 -- 172.21.15.32:0/2655710057 wait complete. 2024-03-16T20:03:40.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:40.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:41.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 Processor -- start 2024-03-16T20:03:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 -- start start 2024-03-16T20:03:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7101542c0 0x7fc710073010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7101546a0 con 0x7fc71015dad0 2024-03-16T20:03:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc710073550 con 0x7fc710070360 2024-03-16T20:03:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc716edc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc710159f20 con 0x7fc7101542c0 2024-03-16T20:03:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc714c78640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7101542c0 0x7fc710073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc70ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc710070360 0x7fc710072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44004/0 (socket says 172.21.15.32:44004) 2024-03-16T20:03:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc715479640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc71015dad0 0x7fc7101648a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58168/0 (socket says 172.21.15.32:58168) 2024-03-16T20:03:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.294+0000 7fc70ffff640 1 -- 172.21.15.32:0/1900839206 learned_addr learned my addr 172.21.15.32:0/1900839206 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.302+0000 7fc70f7fe640 1 -- 172.21.15.32:0/1900839206 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638484867 0 0) 0x7fc710073550 con 0x7fc710070360 2024-03-16T20:03:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.302+0000 7fc70f7fe640 1 -- 172.21.15.32:0/1900839206 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc700003530 con 0x7fc710070360 2024-03-16T20:03:41.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.302+0000 7fc714c78640 1 -- 172.21.15.32:0/1900839206 >> v1:172.21.15.59:6789/0 conn(0x7fc710070360 legacy=0x7fc710072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.302+0000 7fc714c78640 1 -- 172.21.15.32:0/1900839206 >> v1:172.21.15.32:6789/0 conn(0x7fc71015dad0 legacy=0x7fc7101648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.302+0000 7fc714c78640 1 -- 172.21.15.32:0/1900839206 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71015a1e0 con 0x7fc7101542c0 2024-03-16T20:03:41.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.306+0000 7fc714c78640 1 --2- 172.21.15.32:0/1900839206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7101542c0 0x7fc710073010 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc7040097c0 tx=0x7fc70401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ccdcebee45aab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:41.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc70f7fe640 1 -- 172.21.15.32:0/1900839206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040a8050 con 0x7fc7101542c0 2024-03-16T20:03:41.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc70f7fe640 1 -- 172.21.15.32:0/1900839206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7040ac5f0 con 0x7fc7101542c0 2024-03-16T20:03:41.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc70f7fe640 1 -- 172.21.15.32:0/1900839206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040ac980 con 0x7fc7101542c0 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 -- 172.21.15.32:0/1900839206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7101542c0 msgr2=0x7fc710073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 --2- 172.21.15.32:0/1900839206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7101542c0 0x7fc710073010 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc7040097c0 tx=0x7fc70401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 -- 172.21.15.32:0/1900839206 shutdown_connections 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 --2- 172.21.15.32:0/1900839206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7101542c0 0x7fc710073010 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 -- 172.21.15.32:0/1900839206 >> 172.21.15.32:0/1900839206 conn(0x7fc71006c0d0 msgr2=0x7fc71006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 -- 172.21.15.32:0/1900839206 shutdown_connections 2024-03-16T20:03:41.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 -- 172.21.15.32:0/1900839206 wait complete. 2024-03-16T20:03:41.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.314+0000 7fc716edc640 1 Processor -- start 2024-03-16T20:03:41.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 -- start start 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc710070360 0x7fc710152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7101542c0 0x7fc710155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101f7850 con 0x7fc71015dad0 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101f7d80 con 0x7fc7101542c0 2024-03-16T20:03:41.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc716edc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101f7aa0 con 0x7fc710070360 2024-03-16T20:03:41.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc715479640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:41.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc715479640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50962/0 (socket says 172.21.15.32:50962) 2024-03-16T20:03:41.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc715479640 1 -- 172.21.15.32:0/2913064369 learned_addr learned my addr 172.21.15.32:0/2913064369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:41.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.318+0000 7fc70ffff640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7101542c0 0x7fc710155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc714c78640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc710070360 0x7fc710152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc715479640 1 -- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc710070360 msgr2=0x7fc710152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc715479640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc710070360 0x7fc710152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc715479640 1 -- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7101542c0 msgr2=0x7fc710155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc715479640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7101542c0 0x7fc710155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.322+0000 7fc715479640 1 -- 172.21.15.32:0/2913064369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704009420 con 0x7fc71015dad0 2024-03-16T20:03:41.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.326+0000 7fc715479640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc70800c700 tx=0x7fc70800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800ce10 con 0x7fc71015dad0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70800a640 con 0x7fc71015dad0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800a9b0 con 0x7fc71015dad0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc710154d60 con 0x7fc71015dad0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc710155320 con 0x7fc71015dad0 2024-03-16T20:03:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.334+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710070b00 con 0x7fc71015dad0 2024-03-16T20:03:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.358+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc70800ab10 con 0x7fc71015dad0 2024-03-16T20:03:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.358+0000 7fc70d7fa640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 0x7fc6f4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.358+0000 7fc714c78640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 0x7fc6f4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.358+0000 7fc714c78640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 0x7fc6f4069350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc7040ae000 tx=0x7fc704008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.642+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc708055220 con 0x7fc71015dad0 2024-03-16T20:03:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.642+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7080514c0 con 0x7fc71015dad0 2024-03-16T20:03:41.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.846+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc710068000 con 0x7fc71015dad0 2024-03-16T20:03:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.918+0000 7fc70d7fa640 1 -- 172.21.15.32:0/2913064369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc708017090 con 0x7fc71015dad0 2024-03-16T20:03:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:41.901 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 msgr2=0x7fc6f4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 0x7fc6f4069350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc7040ae000 tx=0x7fc704008040 comp rx=0 tx=0).stop 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 msgr2=0x7fc7101562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc70800c700 tx=0x7fc70800cbc0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 shutdown_connections 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc6f4066e90 0x7fc6f4069350 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc71015dad0 0x7fc7101562f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7101542c0 0x7fc710155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 --2- 172.21.15.32:0/2913064369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc710070360 0x7fc710152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 >> 172.21.15.32:0/2913064369 conn(0x7fc71006c0d0 msgr2=0x7fc71014a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 shutdown_connections 2024-03-16T20:03:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:41.922+0000 7fc716edc640 1 -- 172.21.15.32:0/2913064369 wait complete. 2024-03-16T20:03:42.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:03:42.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:03:42.529 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:42.529 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:43.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 Processor -- start 2024-03-16T20:03:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 -- start start 2024-03-16T20:03:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140cdb80 0x7f181406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18140cdf60 0x7f1814158340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1814157eb0 con 0x7f181406f8b0 2024-03-16T20:03:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18140c4c80 con 0x7f18140cdf60 2024-03-16T20:03:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.658+0000 7f181c45d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18140c4f40 con 0x7f18140cdb80 2024-03-16T20:03:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.662+0000 7f181a1f9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140cdb80 0x7f181406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.662+0000 7f18199f8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f181406f8b0 0x7f1814151b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44014/0 (socket says 172.21.15.32:44014) 2024-03-16T20:03:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.662+0000 7f18199f8640 1 -- 172.21.15.32:0/3454066035 learned_addr learned my addr 172.21.15.32:0/3454066035 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.662+0000 7f181a9fa640 1 --2- 172.21.15.32:0/3454066035 >> v2:172.21.15.32:3300/0 conn(0x7f18140cdf60 0x7f1814158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.666+0000 7f181a9fa640 1 -- 172.21.15.32:0/3454066035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140cdb80 msgr2=0x7f181406f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.666+0000 7f181a9fa640 1 --2- 172.21.15.32:0/3454066035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140cdb80 0x7f181406f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.666+0000 7f181a9fa640 1 -- 172.21.15.32:0/3454066035 >> v1:172.21.15.59:6789/0 conn(0x7f181406f8b0 legacy=0x7f1814151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.666+0000 7f181a9fa640 1 -- 172.21.15.32:0/3454066035 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140c5220 con 0x7f18140cdf60 2024-03-16T20:03:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.670+0000 7f181a9fa640 1 --2- 172.21.15.32:0/3454066035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdf60 0x7f1814158340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f180c009d90 tx=0x7f180c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88213e015c58c677 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f18191f7640 1 -- 172.21.15.32:0/3454066035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c0a5a20 con 0x7f18140cdf60 2024-03-16T20:03:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f18191f7640 1 -- 172.21.15.32:0/3454066035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c0a5b80 con 0x7f18140cdf60 2024-03-16T20:03:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f18191f7640 1 -- 172.21.15.32:0/3454066035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c0ad440 con 0x7f18140cdf60 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 -- 172.21.15.32:0/3454066035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdf60 msgr2=0x7f1814158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 --2- 172.21.15.32:0/3454066035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdf60 0x7f1814158340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f180c009d90 tx=0x7f180c01c940 comp rx=0 tx=0).stop 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 -- 172.21.15.32:0/3454066035 shutdown_connections 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 --2- 172.21.15.32:0/3454066035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdf60 0x7f1814158340 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 --2- 172.21.15.32:0/3454066035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140cdb80 0x7f181406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 -- 172.21.15.32:0/3454066035 >> 172.21.15.32:0/3454066035 conn(0x7f1814066ea0 msgr2=0x7f18140672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 -- 172.21.15.32:0/3454066035 shutdown_connections 2024-03-16T20:03:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.674+0000 7f181c45d640 1 -- 172.21.15.32:0/3454066035 wait complete. 2024-03-16T20:03:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 Processor -- start 2024-03-16T20:03:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 -- start start 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18140cdf60 0x7f181414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140ca630 0x7f18141f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181414e210 con 0x7f18140cdb80 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181406cad0 con 0x7f18140cdf60 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181c45d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181406c530 con 0x7f18140ca630 2024-03-16T20:03:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181a1f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181a1f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50974/0 (socket says 172.21.15.32:50974) 2024-03-16T20:03:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181a1f9640 1 -- 172.21.15.32:0/3386118588 learned_addr learned my addr 172.21.15.32:0/3386118588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f181a9fa640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140ca630 0x7f18141f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.678+0000 7f18199f8640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18140cdf60 0x7f181414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 -- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140ca630 msgr2=0x7f18141f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140ca630 0x7f18141f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 -- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18140cdf60 msgr2=0x7f181414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18140cdf60 0x7f181414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 -- 172.21.15.32:0/3386118588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c009a70 con 0x7f18140cdb80 2024-03-16T20:03:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.686+0000 7f181a1f9640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f181000e930 tx=0x7f181000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810009690 con 0x7f18140cdb80 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181000c9a0 con 0x7f18140cdb80 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000cd10 con 0x7f18140cdb80 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181406c7f0 con 0x7f18140cdb80 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1814148ec0 con 0x7f18140cdb80 2024-03-16T20:03:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.694+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1814068000 con 0x7f18140cdb80 2024-03-16T20:03:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.970+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1810013470 con 0x7f18140cdb80 2024-03-16T20:03:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.974+0000 7f180affd640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 0x7f17f8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.974+0000 7f18199f8640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 0x7f17f8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.974+0000 7f18199f8640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 0x7f17f8069380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f181406e820 tx=0x7f1804007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.990+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f18100560a0 con 0x7f18140cdb80 2024-03-16T20:03:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:43.990+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1810052340 con 0x7f18140cdb80 2024-03-16T20:03:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.194+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1814155b60 con 0x7f18140cdb80 2024-03-16T20:03:44.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.218+0000 7f180affd640 1 -- 172.21.15.32:0/3386118588 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1810021090 con 0x7f18140cdb80 2024-03-16T20:03:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:44.199 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:44.199 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:44.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.218+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 msgr2=0x7f17f8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.218+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 0x7f17f8069380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f181406e820 tx=0x7f1804007e90 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 msgr2=0x7f181414e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f181000e930 tx=0x7f181000edf0 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 shutdown_connections 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f17f8066ec0 0x7f17f8069380 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18140ca630 0x7f18141f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18140cdf60 0x7f181414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 --2- 172.21.15.32:0/3386118588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18140cdb80 0x7f181414e800 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 >> 172.21.15.32:0/3386118588 conn(0x7f1814066ea0 msgr2=0x7f1814067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 shutdown_connections 2024-03-16T20:03:44.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:44.222+0000 7f181c45d640 1 -- 172.21.15.32:0/3386118588 wait complete. 2024-03-16T20:03:44.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:44.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:45.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 Processor -- start 2024-03-16T20:03:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 -- start start 2024-03-16T20:03:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0d580c6040 0x7f0d580c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d58070760 con 0x7f0d58158390 2024-03-16T20:03:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d580731c0 con 0x7f0d580c6420 2024-03-16T20:03:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d5dd6e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d580734a0 con 0x7f0d580c6040 2024-03-16T20:03:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d57fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0d58158390 0x7f0d581516d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58182/0 (socket says 172.21.15.32:58182) 2024-03-16T20:03:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d57fff640 1 -- 172.21.15.32:0/3920942716 learned_addr learned my addr 172.21.15.32:0/3920942716 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.958+0000 7f0d577fe640 1 --2- 172.21.15.32:0/3920942716 >> v2:172.21.15.59:3300/0 conn(0x7f0d580c6040 0x7f0d580c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:45.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.962+0000 7f0d567fc640 1 -- 172.21.15.32:0/3920942716 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184747150 0 0) 0x7f0d58070760 con 0x7f0d58158390 2024-03-16T20:03:45.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.962+0000 7f0d567fc640 1 -- 172.21.15.32:0/3920942716 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40003530 con 0x7f0d58158390 2024-03-16T20:03:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.966+0000 7f0d577fe640 1 -- 172.21.15.32:0/3920942716 >> v1:172.21.15.59:6789/0 conn(0x7f0d580c6420 legacy=0x7f0d58157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.966+0000 7f0d577fe640 1 -- 172.21.15.32:0/3920942716 >> v1:172.21.15.32:6789/0 conn(0x7f0d58158390 legacy=0x7f0d581516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.966+0000 7f0d577fe640 1 -- 172.21.15.32:0/3920942716 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5806fe30 con 0x7f0d580c6040 2024-03-16T20:03:45.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.970+0000 7f0d577fe640 1 --2- 172.21.15.32:0/3920942716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d580c6040 0x7f0d580c8800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d480097c0 tx=0x7f0d4801c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c751a9886781533 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.978+0000 7f0d567fc640 1 -- 172.21.15.32:0/3920942716 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480a7070 con 0x7f0d580c6040 2024-03-16T20:03:45.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.978+0000 7f0d567fc640 1 -- 172.21.15.32:0/3920942716 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d480ab430 con 0x7f0d580c6040 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/3920942716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d580c6040 msgr2=0x7f0d580c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/3920942716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d580c6040 0x7f0d580c8800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d480097c0 tx=0x7f0d4801c430 comp rx=0 tx=0).stop 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/3920942716 shutdown_connections 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/3920942716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d580c6040 0x7f0d580c8800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/3920942716 >> 172.21.15.32:0/3920942716 conn(0x7f0d5806c0d0 msgr2=0x7f0d5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/3920942716 shutdown_connections 2024-03-16T20:03:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/3920942716 wait complete. 2024-03-16T20:03:45.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 Processor -- start 2024-03-16T20:03:45.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- start start 2024-03-16T20:03:45.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 0x7f0d581577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:45.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 0x7f0d581590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:45.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d58158390 0x7f0d581f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d58157fd0 con 0x7f0d580c6040 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d581456a0 con 0x7f0d58158390 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.982+0000 7f0d5dd6e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d58157d10 con 0x7f0d580c6420 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.986+0000 7f0d56ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 0x7f0d581590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.986+0000 7f0d577fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 0x7f0d581577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.986+0000 7f0d56ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 0x7f0d581590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:43024/0 (socket says 172.21.15.32:43024) 2024-03-16T20:03:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.986+0000 7f0d56ffd640 1 -- 172.21.15.32:0/2698068149 learned_addr learned my addr 172.21.15.32:0/2698068149 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:45.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.986+0000 7f0d57fff640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d58158390 0x7f0d581f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:45.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.990+0000 7f0d577fe640 1 -- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 msgr2=0x7f0d581590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:45.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.990+0000 7f0d577fe640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 0x7f0d581590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:45.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.990+0000 7f0d577fe640 1 -- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d58158390 msgr2=0x7f0d581f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:45.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.990+0000 7f0d577fe640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d58158390 0x7f0d581f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:45.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.990+0000 7f0d577fe640 1 -- 172.21.15.32:0/2698068149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d48009420 con 0x7f0d580c6040 2024-03-16T20:03:45.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.994+0000 7f0d577fe640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 0x7f0d581577d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0d480ac000 tx=0x7f0d480191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.998+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48003a30 con 0x7f0d580c6040 2024-03-16T20:03:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.998+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48003b90 con 0x7f0d580c6040 2024-03-16T20:03:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.998+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d58145940 con 0x7f0d580c6040 2024-03-16T20:03:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.998+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480044e0 con 0x7f0d580c6040 2024-03-16T20:03:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:45.998+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d58145f00 con 0x7f0d580c6040 2024-03-16T20:03:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.002+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d581f42c0 con 0x7f0d580c6040 2024-03-16T20:03:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.026+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d480ab5a0 con 0x7f0d580c6040 2024-03-16T20:03:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.026+0000 7f0d3ffff640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 0x7f0d44069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.026+0000 7f0d56ffd640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 0x7f0d44069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:46.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.026+0000 7f0d56ffd640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 0x7f0d44069350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d58072980 tx=0x7f0d4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.026+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0d480ef690 con 0x7f0d580c6040 2024-03-16T20:03:46.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.358+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0d480eb930 con 0x7f0d580c6040 2024-03-16T20:03:46.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.622+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d580c7d40 con 0x7f0d580c6040 2024-03-16T20:03:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.650+0000 7f0d3ffff640 1 -- 172.21.15.32:0/2698068149 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f0d480eeff0 con 0x7f0d580c6040 2024-03-16T20:03:46.634 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:46.635 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:46.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 msgr2=0x7f0d44069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 0x7f0d44069350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d58072980 tx=0x7f0d4c009040 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 msgr2=0x7f0d581577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 0x7f0d581577d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0d480ac000 tx=0x7f0d480191d0 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 shutdown_connections 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d44066e90 0x7f0d44069350 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d58158390 0x7f0d581f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d580c6420 0x7f0d581590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 --2- 172.21.15.32:0/2698068149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d580c6040 0x7f0d581577d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 >> 172.21.15.32:0/2698068149 conn(0x7f0d5806c0d0 msgr2=0x7f0d5814f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 shutdown_connections 2024-03-16T20:03:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:46.654+0000 7f0d5dd6e640 1 -- 172.21.15.32:0/2698068149 wait complete. 2024-03-16T20:03:47.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:47.266 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:48.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:48.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 Processor -- start 2024-03-16T20:03:48.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 -- start start 2024-03-16T20:03:48.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6240c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:48.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6240c8ec0 con 0x7fe624151000 2024-03-16T20:03:48.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe624072950 con 0x7fe6240c36b0 2024-03-16T20:03:48.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe62b3e9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe624072f10 con 0x7fe6240c8980 2024-03-16T20:03:48.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe629986640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6240c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:48.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe628984640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe6240c36b0 0x7fe62415f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44022/0 (socket says 172.21.15.32:44022) 2024-03-16T20:03:48.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.394+0000 7fe628984640 1 -- 172.21.15.32:0/1699936181 learned_addr learned my addr 172.21.15.32:0/1699936181 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.398+0000 7fe61bfff640 1 -- 172.21.15.32:0/1699936181 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958970783 0 0) 0x7fe624072950 con 0x7fe6240c36b0 2024-03-16T20:03:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.398+0000 7fe61bfff640 1 -- 172.21.15.32:0/1699936181 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe610003530 con 0x7fe6240c36b0 2024-03-16T20:03:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.402+0000 7fe629986640 1 -- 172.21.15.32:0/1699936181 >> v1:172.21.15.59:6789/0 conn(0x7fe6240c36b0 legacy=0x7fe62415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.402+0000 7fe629986640 1 -- 172.21.15.32:0/1699936181 >> v1:172.21.15.32:6789/0 conn(0x7fe624151000 legacy=0x7fe6240c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.402+0000 7fe629986640 1 -- 172.21.15.32:0/1699936181 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624072c30 con 0x7fe6240c8980 2024-03-16T20:03:48.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.410+0000 7fe629986640 1 --2- 172.21.15.32:0/1699936181 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6240c8420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009dc0 tx=0x7fe61c01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27e56ff6709e2cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe61bfff640 1 -- 172.21.15.32:0/1699936181 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c0a5900 con 0x7fe6240c8980 2024-03-16T20:03:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe61bfff640 1 -- 172.21.15.32:0/1699936181 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c0a5a60 con 0x7fe6240c8980 2024-03-16T20:03:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe61bfff640 1 -- 172.21.15.32:0/1699936181 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c0a5df0 con 0x7fe6240c8980 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1699936181 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 msgr2=0x7fe6240c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1699936181 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6240c8420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009dc0 tx=0x7fe61c01c950 comp rx=0 tx=0).stop 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1699936181 shutdown_connections 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1699936181 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6240c8420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1699936181 >> 172.21.15.32:0/1699936181 conn(0x7fe62406c0d0 msgr2=0x7fe62406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1699936181 shutdown_connections 2024-03-16T20:03:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.418+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1699936181 wait complete. 2024-03-16T20:03:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 Processor -- start 2024-03-16T20:03:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 -- start start 2024-03-16T20:03:48.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:48.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6241587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:48.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe624151000 0x7fe624149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:48.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241f3aa0 con 0x7fe6240c36b0 2024-03-16T20:03:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241f61d0 con 0x7fe624151000 2024-03-16T20:03:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe62b3e9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241f64b0 con 0x7fe6240c8980 2024-03-16T20:03:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.422+0000 7fe629185640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.426+0000 7fe628984640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6241587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.426+0000 7fe629986640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe624151000 0x7fe624149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.426+0000 7fe629185640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51004/0 (socket says 172.21.15.32:51004) 2024-03-16T20:03:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.426+0000 7fe629185640 1 -- 172.21.15.32:0/1001290171 learned_addr learned my addr 172.21.15.32:0/1001290171 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.430+0000 7fe629185640 1 -- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 msgr2=0x7fe6241587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.430+0000 7fe629185640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6241587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.430+0000 7fe629185640 1 -- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe624151000 msgr2=0x7fe624149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.430+0000 7fe629185640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe624151000 0x7fe624149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:48.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.430+0000 7fe629185640 1 -- 172.21.15.32:0/1001290171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c009a70 con 0x7fe6240c36b0 2024-03-16T20:03:48.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.434+0000 7fe629185640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe62000b3e0 tx=0x7fe62000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.438+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620011830 con 0x7fe6240c36b0 2024-03-16T20:03:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.438+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6200119b0 con 0x7fe6240c36b0 2024-03-16T20:03:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.438+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6241f6790 con 0x7fe6240c36b0 2024-03-16T20:03:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.442+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620011d60 con 0x7fe6240c36b0 2024-03-16T20:03:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.442+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6241f6d50 con 0x7fe6240c36b0 2024-03-16T20:03:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.442+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe624068000 con 0x7fe6240c36b0 2024-03-16T20:03:48.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.474+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe62001a400 con 0x7fe6240c36b0 2024-03-16T20:03:48.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.478+0000 7fe619ffb640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 0x7fe608069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:48.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.478+0000 7fe628984640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 0x7fe608069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:48.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.478+0000 7fe628984640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 0x7fe608069380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe62406c6b0 tx=0x7fe614007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:48.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.770+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe620055c00 con 0x7fe6240c36b0 2024-03-16T20:03:48.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.770+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe620051ea0 con 0x7fe6240c36b0 2024-03-16T20:03:48.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:48.982+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe624073410 con 0x7fe6240c36b0 2024-03-16T20:03:49.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.026+0000 7fe619ffb640 1 -- 172.21.15.32:0/1001290171 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe620015090 con 0x7fe6240c36b0 2024-03-16T20:03:49.009 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:49.009 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 msgr2=0x7fe608069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 0x7fe608069380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe62406c6b0 tx=0x7fe614007d40 comp rx=0 tx=0).stop 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 msgr2=0x7fe624148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe62000b3e0 tx=0x7fe62000b8a0 comp rx=0 tx=0).stop 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 shutdown_connections 2024-03-16T20:03:49.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe608066ec0 0x7fe608069380 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe624151000 0x7fe624149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe6240c8980 0x7fe6241587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 --2- 172.21.15.32:0/1001290171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6240c36b0 0x7fe624148ac0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 >> 172.21.15.32:0/1001290171 conn(0x7fe62406c0d0 msgr2=0x7fe624159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 shutdown_connections 2024-03-16T20:03:49.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:49.030+0000 7fe62b3e9640 1 -- 172.21.15.32:0/1001290171 wait complete. 2024-03-16T20:03:49.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:49.633 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:50.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 Processor -- start 2024-03-16T20:03:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 -- start start 2024-03-16T20:03:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7eff9806e220 0x7eff98165860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9815a390 con 0x7eff980c73f0 2024-03-16T20:03:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98072bd0 con 0x7eff9806de40 2024-03-16T20:03:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.762+0000 7eff9d67b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff98071b70 con 0x7eff9806e220 2024-03-16T20:03:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.766+0000 7eff967fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7eff980c73f0 0x7eff98153e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36014/0 (socket says 172.21.15.32:36014) 2024-03-16T20:03:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.766+0000 7eff967fc640 1 -- 172.21.15.32:0/3071903810 learned_addr learned my addr 172.21.15.32:0/3071903810 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.766+0000 7eff977fe640 1 --2- 172.21.15.32:0/3071903810 >> v2:172.21.15.59:3300/0 conn(0x7eff9806e220 0x7eff98165860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:50.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.774+0000 7eff95ffb640 1 -- 172.21.15.32:0/3071903810 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076730317 0 0) 0x7eff9815a390 con 0x7eff980c73f0 2024-03-16T20:03:50.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.774+0000 7eff95ffb640 1 -- 172.21.15.32:0/3071903810 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80003530 con 0x7eff980c73f0 2024-03-16T20:03:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.774+0000 7eff977fe640 1 -- 172.21.15.32:0/3071903810 >> v1:172.21.15.59:6789/0 conn(0x7eff9806de40 legacy=0x7eff9815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.774+0000 7eff977fe640 1 -- 172.21.15.32:0/3071903810 >> v1:172.21.15.32:6789/0 conn(0x7eff980c73f0 legacy=0x7eff98153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.774+0000 7eff977fe640 1 -- 172.21.15.32:0/3071903810 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98072eb0 con 0x7eff9806e220 2024-03-16T20:03:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.782+0000 7eff977fe640 1 --2- 172.21.15.32:0/3071903810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff98165860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff90009d90 tx=0x7eff9001f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79012ef3155faf9b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff95ffb640 1 -- 172.21.15.32:0/3071903810 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900af600 con 0x7eff9806e220 2024-03-16T20:03:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff95ffb640 1 -- 172.21.15.32:0/3071903810 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900ab050 con 0x7eff9806e220 2024-03-16T20:03:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff95ffb640 1 -- 172.21.15.32:0/3071903810 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900af980 con 0x7eff9806e220 2024-03-16T20:03:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 -- 172.21.15.32:0/3071903810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 msgr2=0x7eff98165860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 --2- 172.21.15.32:0/3071903810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff98165860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff90009d90 tx=0x7eff9001f120 comp rx=0 tx=0).stop 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 -- 172.21.15.32:0/3071903810 shutdown_connections 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 --2- 172.21.15.32:0/3071903810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff98165860 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 -- 172.21.15.32:0/3071903810 >> 172.21.15.32:0/3071903810 conn(0x7eff980cbfe0 msgr2=0x7eff980cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 -- 172.21.15.32:0/3071903810 shutdown_connections 2024-03-16T20:03:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.790+0000 7eff9d67b640 1 -- 172.21.15.32:0/3071903810 wait complete. 2024-03-16T20:03:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 Processor -- start 2024-03-16T20:03:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 -- start start 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff980c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff981eff50 0x7eff980c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981f0500 con 0x7eff9806de40 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff980c3b40 con 0x7eff9806e220 2024-03-16T20:03:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff9d67b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff980c3e20 con 0x7eff981eff50 2024-03-16T20:03:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff96ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff96ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35568/0 (socket says 172.21.15.32:35568) 2024-03-16T20:03:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff96ffd640 1 -- 172.21.15.32:0/1279877684 learned_addr learned my addr 172.21.15.32:0/1279877684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.794+0000 7eff977fe640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff981eff50 0x7eff980c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:50.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.798+0000 7eff967fc640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff980c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 -- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff981eff50 msgr2=0x7eff980c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff981eff50 0x7eff980c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 -- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 msgr2=0x7eff980c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff980c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:50.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 -- 172.21.15.32:0/1279877684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90009a70 con 0x7eff9806de40 2024-03-16T20:03:50.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.802+0000 7eff96ffd640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c490 tx=0x7eff88001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88003600 con 0x7eff9806de40 2024-03-16T20:03:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff88003780 con 0x7eff9806de40 2024-03-16T20:03:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff9d67b640 1 -- 172.21.15.32:0/1279877684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff980c4100 con 0x7eff9806de40 2024-03-16T20:03:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88003b30 con 0x7eff9806de40 2024-03-16T20:03:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff9d67b640 1 -- 172.21.15.32:0/1279877684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff980c46c0 con 0x7eff9806de40 2024-03-16T20:03:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:50.810+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff700052a0 con 0x7eff9806de40 2024-03-16T20:03:51.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.090+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7eff88003c90 con 0x7eff9806de40 2024-03-16T20:03:51.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.090+0000 7eff7f7fe640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 0x7eff84069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:51.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.090+0000 7eff967fc640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 0x7eff84069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:51.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.090+0000 7eff967fc640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 0x7eff84069300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7eff9806c250 tx=0x7eff8c007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:51.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.110+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7eff88054a90 con 0x7eff9806de40 2024-03-16T20:03:51.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.110+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7eff88050d30 con 0x7eff9806de40 2024-03-16T20:03:51.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.334+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff70005900 con 0x7eff9806de40 2024-03-16T20:03:51.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.402+0000 7eff7f7fe640 1 -- 172.21.15.32:0/1279877684 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7eff70005900 con 0x7eff9806de40 2024-03-16T20:03:51.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:51.384 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:51.385 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 msgr2=0x7eff84069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 0x7eff84069300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7eff9806c250 tx=0x7eff8c007d40 comp rx=0 tx=0).stop 2024-03-16T20:03:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 msgr2=0x7eff980b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c490 tx=0x7eff88001e80 comp rx=0 tx=0).stop 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 shutdown_connections 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7eff84066e40 0x7eff84069300 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7eff981eff50 0x7eff980c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff9806e220 0x7eff980c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 --2- 172.21.15.32:0/1279877684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff9806de40 0x7eff980b0310 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 >> 172.21.15.32:0/1279877684 conn(0x7eff980cbfe0 msgr2=0x7eff9806b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 shutdown_connections 2024-03-16T20:03:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:51.406+0000 7eff7d7fa640 1 -- 172.21.15.32:0/1279877684 wait complete. 2024-03-16T20:03:52.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:52.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:53.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:53.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 Processor -- start 2024-03-16T20:03:53.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 -- start start 2024-03-16T20:03:53.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec44150f90 0x7fec44156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fec440c87f0 0x7fec441616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440700f0 con 0x7fec440c6fd0 2024-03-16T20:03:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec44072e70 con 0x7fec44150f90 2024-03-16T20:03:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.142+0000 7fec49bf1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec440728b0 con 0x7fec440c87f0 2024-03-16T20:03:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec3bfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fec440c6fd0 0x7fec4406c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36024/0 (socket says 172.21.15.32:36024) 2024-03-16T20:03:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec3bfff640 1 -- 172.21.15.32:0/2393405303 learned_addr learned my addr 172.21.15.32:0/2393405303 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec437fe640 1 --2- 172.21.15.32:0/2393405303 >> v2:172.21.15.32:3300/0 conn(0x7fec44150f90 0x7fec44156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec43fff640 1 --2- 172.21.15.32:0/2393405303 >> v2:172.21.15.59:3300/0 conn(0x7fec440c87f0 0x7fec441616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec42ffd640 1 -- 172.21.15.32:0/2393405303 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053914649 0 0) 0x7fec440700f0 con 0x7fec440c6fd0 2024-03-16T20:03:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.146+0000 7fec42ffd640 1 -- 172.21.15.32:0/2393405303 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30003530 con 0x7fec440c6fd0 2024-03-16T20:03:53.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.150+0000 7fec437fe640 1 -- 172.21.15.32:0/2393405303 >> v2:172.21.15.59:3300/0 conn(0x7fec440c87f0 msgr2=0x7fec441616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:53.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.150+0000 7fec437fe640 1 --2- 172.21.15.32:0/2393405303 >> v2:172.21.15.59:3300/0 conn(0x7fec440c87f0 0x7fec441616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.150+0000 7fec437fe640 1 -- 172.21.15.32:0/2393405303 >> v1:172.21.15.32:6789/0 conn(0x7fec440c6fd0 legacy=0x7fec4406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.150+0000 7fec437fe640 1 -- 172.21.15.32:0/2393405303 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44072b90 con 0x7fec44150f90 2024-03-16T20:03:53.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.158+0000 7fec437fe640 1 --2- 172.21.15.32:0/2393405303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec44150f90 0x7fec44156d30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec4406f0a0 tx=0x7fec3401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2709cd9ccd0d3014 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec42ffd640 1 -- 172.21.15.32:0/2393405303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340ab680 con 0x7fec44150f90 2024-03-16T20:03:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec42ffd640 1 -- 172.21.15.32:0/2393405303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec340ab7e0 con 0x7fec44150f90 2024-03-16T20:03:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec42ffd640 1 -- 172.21.15.32:0/2393405303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340abb70 con 0x7fec44150f90 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- 172.21.15.32:0/2393405303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec44150f90 msgr2=0x7fec44156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 --2- 172.21.15.32:0/2393405303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec44150f90 0x7fec44156d30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec4406f0a0 tx=0x7fec3401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- 172.21.15.32:0/2393405303 shutdown_connections 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 --2- 172.21.15.32:0/2393405303 >> v2:172.21.15.59:3300/0 conn(0x7fec440c87f0 0x7fec441616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 --2- 172.21.15.32:0/2393405303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec44150f90 0x7fec44156d30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- 172.21.15.32:0/2393405303 >> 172.21.15.32:0/2393405303 conn(0x7fec440cc7a0 msgr2=0x7fec440ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- 172.21.15.32:0/2393405303 shutdown_connections 2024-03-16T20:03:53.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- 172.21.15.32:0/2393405303 wait complete. 2024-03-16T20:03:53.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 Processor -- start 2024-03-16T20:03:53.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.166+0000 7fec49bf1640 1 -- start start 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec440c87f0 0x7fec4415f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec44150f90 0x7fec441424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec441f7690 con 0x7fec440c6fd0 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec44156c50 con 0x7fec440c87f0 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec49bf1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec441f6a80 con 0x7fec44150f90 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec437fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec437fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35590/0 (socket says 172.21.15.32:35590) 2024-03-16T20:03:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec437fe640 1 -- 172.21.15.32:0/1824931953 learned_addr learned my addr 172.21.15.32:0/1824931953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec43fff640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec44150f90 0x7fec441424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.170+0000 7fec3bfff640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec440c87f0 0x7fec4415f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.174+0000 7fec437fe640 1 -- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec44150f90 msgr2=0x7fec441424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.174+0000 7fec437fe640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec44150f90 0x7fec441424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.174+0000 7fec437fe640 1 -- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec440c87f0 msgr2=0x7fec4415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.174+0000 7fec437fe640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec440c87f0 0x7fec4415f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.174+0000 7fec437fe640 1 -- 172.21.15.32:0/1824931953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34009420 con 0x7fec440c6fd0 2024-03-16T20:03:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.178+0000 7fec437fe640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec340ad040 tx=0x7fec3401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340bf610 con 0x7fec440c6fd0 2024-03-16T20:03:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec340bf770 con 0x7fec440c6fd0 2024-03-16T20:03:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec441f6d60 con 0x7fec440c6fd0 2024-03-16T20:03:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340bfb20 con 0x7fec440c6fd0 2024-03-16T20:03:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec441f7320 con 0x7fec440c6fd0 2024-03-16T20:03:53.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.182+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4415b2c0 con 0x7fec440c6fd0 2024-03-16T20:03:53.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.470+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec340bfc80 con 0x7fec440c6fd0 2024-03-16T20:03:53.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.470+0000 7fec40ff9640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 0x7fec24069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:53.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.470+0000 7fec3bfff640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 0x7fec24069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:53.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.470+0000 7fec3bfff640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 0x7fec24069380 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec2c007fd0 tx=0x7fec2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:53.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.498+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fec340f06f0 con 0x7fec440c6fd0 2024-03-16T20:03:53.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.498+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fec340ec990 con 0x7fec440c6fd0 2024-03-16T20:03:53.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.694+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec44068000 con 0x7fec440c6fd0 2024-03-16T20:03:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.718+0000 7fec40ff9640 1 -- 172.21.15.32:0/1824931953 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fec340f0050 con 0x7fec440c6fd0 2024-03-16T20:03:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:53.699 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:53.700 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:53.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 msgr2=0x7fec24069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 0x7fec24069380 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec2c007fd0 tx=0x7fec2c009040 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 msgr2=0x7fec440cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec340ad040 tx=0x7fec3401ace0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 shutdown_connections 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec24066ec0 0x7fec24069380 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec44150f90 0x7fec441424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec440c87f0 0x7fec4415f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 --2- 172.21.15.32:0/1824931953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec440c6fd0 0x7fec440cb880 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 >> 172.21.15.32:0/1824931953 conn(0x7fec440cc7a0 msgr2=0x7fec4406b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 shutdown_connections 2024-03-16T20:03:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:53.722+0000 7fec49bf1640 1 -- 172.21.15.32:0/1824931953 wait complete. 2024-03-16T20:03:54.333 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:54.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:55.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:55.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 Processor -- start 2024-03-16T20:03:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 -- start start 2024-03-16T20:03:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02e00c3e30 0x7f02e0150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02e00c4210 0x7f02e0151050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00c3940 con 0x7f02e0072ed0 2024-03-16T20:03:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e00c7ca0 con 0x7f02e00c4210 2024-03-16T20:03:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02e51ec640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e0072cc0 con 0x7f02e00c3e30 2024-03-16T20:03:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02de59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02e00c4210 0x7f02e0151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02de59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02e00c4210 0x7f02e0151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35602/0 (socket says 172.21.15.32:35602) 2024-03-16T20:03:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02de59c640 1 -- 172.21.15.32:0/2750835621 learned_addr learned my addr 172.21.15.32:0/2750835621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.462+0000 7f02ded9d640 1 --2- 172.21.15.32:0/2750835621 >> v2:172.21.15.59:3300/0 conn(0x7f02e00c3e30 0x7f02e0150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.466+0000 7f02de59c640 1 -- 172.21.15.32:0/2750835621 >> v2:172.21.15.59:3300/0 conn(0x7f02e00c3e30 msgr2=0x7f02e0150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.466+0000 7f02de59c640 1 --2- 172.21.15.32:0/2750835621 >> v2:172.21.15.59:3300/0 conn(0x7f02e00c3e30 0x7f02e0150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.466+0000 7f02de59c640 1 -- 172.21.15.32:0/2750835621 >> v1:172.21.15.59:6789/0 conn(0x7f02e0072ed0 legacy=0x7f02e00c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.466+0000 7f02de59c640 1 -- 172.21.15.32:0/2750835621 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e00c7780 con 0x7f02e00c4210 2024-03-16T20:03:55.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.470+0000 7f02de59c640 1 --2- 172.21.15.32:0/2750835621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e00c4210 0x7f02e0151050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f02e00ca1e0 tx=0x7f02d401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499166e12bae3219 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.474+0000 7f02ddd9b640 1 -- 172.21.15.32:0/2750835621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40ad640 con 0x7f02e00c4210 2024-03-16T20:03:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02ddd9b640 1 -- 172.21.15.32:0/2750835621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40ad7a0 con 0x7f02e00c4210 2024-03-16T20:03:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02ddd9b640 1 -- 172.21.15.32:0/2750835621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40adb30 con 0x7f02e00c4210 2024-03-16T20:03:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 -- 172.21.15.32:0/2750835621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e00c4210 msgr2=0x7f02e0151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 --2- 172.21.15.32:0/2750835621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e00c4210 0x7f02e0151050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f02e00ca1e0 tx=0x7f02d401c4e0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 -- 172.21.15.32:0/2750835621 shutdown_connections 2024-03-16T20:03:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 --2- 172.21.15.32:0/2750835621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e00c4210 0x7f02e0151050 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 --2- 172.21.15.32:0/2750835621 >> v2:172.21.15.59:3300/0 conn(0x7f02e00c3e30 0x7f02e0150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 -- 172.21.15.32:0/2750835621 >> 172.21.15.32:0/2750835621 conn(0x7f02e006e6c0 msgr2=0x7f02e006c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 -- 172.21.15.32:0/2750835621 shutdown_connections 2024-03-16T20:03:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.478+0000 7f02e51ec640 1 -- 172.21.15.32:0/2750835621 wait complete. 2024-03-16T20:03:55.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 Processor -- start 2024-03-16T20:03:55.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 -- start start 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e00c3e30 0x7f02e01535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e00c4210 0x7f02e0154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e0155910 con 0x7f02e0072ed0 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01548f0 con 0x7f02e00c4210 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02e51ec640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01f4820 con 0x7f02e00c3e30 2024-03-16T20:03:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02ded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02ded9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35612/0 (socket says 172.21.15.32:35612) 2024-03-16T20:03:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02ded9d640 1 -- 172.21.15.32:0/285196142 learned_addr learned my addr 172.21.15.32:0/285196142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02de59c640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e00c3e30 0x7f02e01535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.482+0000 7f02df59e640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e00c4210 0x7f02e0154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.490+0000 7f02ded9d640 1 -- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e00c3e30 msgr2=0x7f02e01535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.490+0000 7f02ded9d640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e00c3e30 0x7f02e01535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.490+0000 7f02ded9d640 1 -- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e00c4210 msgr2=0x7f02e0154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.490+0000 7f02ded9d640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e00c4210 0x7f02e0154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.490+0000 7f02ded9d640 1 -- 172.21.15.32:0/285196142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40094d0 con 0x7f02e0072ed0 2024-03-16T20:03:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.494+0000 7f02ded9d640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f02d000e930 tx=0x7f02d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.498+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0009690 con 0x7f02e0072ed0 2024-03-16T20:03:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.498+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d000ca00 con 0x7f02e0072ed0 2024-03-16T20:03:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.498+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e014d1f0 con 0x7f02e0072ed0 2024-03-16T20:03:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.498+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000cd70 con 0x7f02e0072ed0 2024-03-16T20:03:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.498+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e01542a0 con 0x7f02e0072ed0 2024-03-16T20:03:55.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.502+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e0068000 con 0x7f02e0072ed0 2024-03-16T20:03:55.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.782+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02d0013470 con 0x7f02e0072ed0 2024-03-16T20:03:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.782+0000 7f02c77fe640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 0x7f02cc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.782+0000 7f02de59c640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 0x7f02cc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.782+0000 7f02de59c640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 0x7f02cc069350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f02d40af000 tx=0x7f02d40a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.810+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f02d00560f0 con 0x7f02e0072ed0 2024-03-16T20:03:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:55.810+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f02d0052390 con 0x7f02e0072ed0 2024-03-16T20:03:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.018+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e00c8570 con 0x7f02e0072ed0 2024-03-16T20:03:56.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.034+0000 7f02c77fe640 1 -- 172.21.15.32:0/285196142 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f02d0021020 con 0x7f02e0072ed0 2024-03-16T20:03:56.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:56.016 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:56.016 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:56.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.034+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 msgr2=0x7f02cc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:56.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.034+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 0x7f02cc069350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f02d40af000 tx=0x7f02d40a7070 comp rx=0 tx=0).stop 2024-03-16T20:03:56.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.034+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 msgr2=0x7f02e01f42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.034+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f02d000e930 tx=0x7f02d000edf0 comp rx=0 tx=0).stop 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 shutdown_connections 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02cc066e90 0x7f02cc069350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e00c4210 0x7f02e0154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e00c3e30 0x7f02e01535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 --2- 172.21.15.32:0/285196142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e0072ed0 0x7f02e01f42e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 >> 172.21.15.32:0/285196142 conn(0x7f02e006e6c0 msgr2=0x7f02e006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 shutdown_connections 2024-03-16T20:03:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:56.038+0000 7f02e51ec640 1 -- 172.21.15.32:0/285196142 wait complete. 2024-03-16T20:03:56.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:56.645 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:57.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:03:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 Processor -- start 2024-03-16T20:03:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 -- start start 2024-03-16T20:03:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60e40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e40c8ec0 con 0x7f60e4151000 2024-03-16T20:03:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e4072950 con 0x7f60e40c8980 2024-03-16T20:03:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.774+0000 7f60eb153640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4072f10 con 0x7f60e40c36b0 2024-03-16T20:03:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.778+0000 7f60e8eef640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60e4151000 0x7f60e40c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36038/0 (socket says 172.21.15.32:36038) 2024-03-16T20:03:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.778+0000 7f60e8eef640 1 -- 172.21.15.32:0/3576547586 learned_addr learned my addr 172.21.15.32:0/3576547586 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:03:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.778+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60e40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.778+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766567799 0 0) 0x7f60e40c8ec0 con 0x7f60e4151000 2024-03-16T20:03:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.778+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d4003530 con 0x7f60e4151000 2024-03-16T20:03:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803832672 0 0) 0x7f60d4003530 con 0x7f60e4151000 2024-03-16T20:03:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 msgr2=0x7f60e40c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60e40c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 >> v1:172.21.15.59:6789/0 conn(0x7f60e40c8980 legacy=0x7f60e41615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4072c30 con 0x7f60e4151000 2024-03-16T20:03:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f60d8001680 con 0x7f60e4151000 2024-03-16T20:03:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 >> v1:172.21.15.32:6789/0 conn(0x7f60e4151000 legacy=0x7f60e40c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d40039b0 0x7f60d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4004310 0x7f60d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60d4004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4072f10 con 0x7f60d4004310 2024-03-16T20:03:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4007b70 con 0x7f60d40039b0 2024-03-16T20:03:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.782+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4007d30 con 0x7f60e40c36b0 2024-03-16T20:03:57.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.790+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4004310 0x7f60d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.790+0000 7f60e96f0640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60d4004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.790+0000 7f60e8eef640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d40039b0 0x7f60d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.794+0000 7f60e3fff640 1 -- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 msgr2=0x7f60d4004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.794+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e40c36b0 0x7f60d4004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.794+0000 7f60e3fff640 1 -- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d40039b0 msgr2=0x7f60d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:03:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.794+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d40039b0 0x7f60d4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.794+0000 7f60e3fff640 1 -- 172.21.15.32:0/3576547586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d4003530 con 0x7f60d4004310 2024-03-16T20:03:57.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.798+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4004310 0x7f60d40047d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009ae0 tx=0x7f60d001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc31789c570854ae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00a5660 con 0x7f60d4004310 2024-03-16T20:03:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d00a57c0 con 0x7f60d4004310 2024-03-16T20:03:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60e37fe640 1 -- 172.21.15.32:0/3576547586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00a5b70 con 0x7f60d4004310 2024-03-16T20:03:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- 172.21.15.32:0/3576547586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4004310 msgr2=0x7f60d40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 --2- 172.21.15.32:0/3576547586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4004310 0x7f60d40047d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009ae0 tx=0x7f60d001ca70 comp rx=0 tx=0).stop 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60e96f0640 1 -- 172.21.15.32:0/3576547586 reap_dead start 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- 172.21.15.32:0/3576547586 shutdown_connections 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- 172.21.15.32:0/3576547586 >> 172.21.15.32:0/3576547586 conn(0x7f60e406c0d0 msgr2=0x7f60e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- 172.21.15.32:0/3576547586 shutdown_connections 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- 172.21.15.32:0/3576547586 wait complete. 2024-03-16T20:03:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 Processor -- start 2024-03-16T20:03:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.802+0000 7f60eb153640 1 -- start start 2024-03-16T20:03:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e41e20a0 0x7f60e4144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e4143500 0x7f60e4142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4145b70 con 0x7f60e41e2480 2024-03-16T20:03:57.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e41444d0 con 0x7f60e4143500 2024-03-16T20:03:57.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60eb153640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40cc350 con 0x7f60e41e20a0 2024-03-16T20:03:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60e8eef640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e41e20a0 0x7f60e4144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35628/0 (socket says 172.21.15.32:35628) 2024-03-16T20:03:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60e3fff640 1 -- 172.21.15.32:0/3717978119 learned_addr learned my addr 172.21.15.32:0/3717978119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:03:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.806+0000 7f60e96f0640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e4143500 0x7f60e4142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.810+0000 7f60e3fff640 1 -- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e41e20a0 msgr2=0x7f60e4144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.810+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e41e20a0 0x7f60e4144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.810+0000 7f60e3fff640 1 -- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e4143500 msgr2=0x7f60e4142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.810+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e4143500 0x7f60e4142c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.810+0000 7f60e3fff640 1 -- 172.21.15.32:0/3717978119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e41e1af0 con 0x7f60e41e2480 2024-03-16T20:03:57.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.814+0000 7f60e3fff640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f60d00b0040 tx=0x7f60d0019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:57.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00a5660 con 0x7f60e41e2480 2024-03-16T20:03:57.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d00b8690 con 0x7f60e41e2480 2024-03-16T20:03:57.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4143190 con 0x7f60e41e2480 2024-03-16T20:03:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00b8a40 con 0x7f60e41e2480 2024-03-16T20:03:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e41e0190 con 0x7f60e41e2480 2024-03-16T20:03:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:57.818+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e4073110 con 0x7f60e41e2480 2024-03-16T20:03:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.098+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60d00b4070 con 0x7f60e41e2480 2024-03-16T20:03:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.098+0000 7f60e17fa640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 0x7f60c8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:03:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.098+0000 7f60e8eef640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 0x7f60c8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:03:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.098+0000 7f60e8eef640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 0x7f60c8069380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f60d8000c10 tx=0x7f60d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:03:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.122+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f60d00f0110 con 0x7f60e41e2480 2024-03-16T20:03:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.122+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f60d00ec3b0 con 0x7f60e41e2480 2024-03-16T20:03:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.326+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60e4073110 con 0x7f60e41e2480 2024-03-16T20:03:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.346+0000 7f60e17fa640 1 -- 172.21.15.32:0/3717978119 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f60e4073110 con 0x7f60e41e2480 2024-03-16T20:03:58.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:03:58.330 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:03:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:03:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 msgr2=0x7f60c8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 0x7f60c8069380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f60d8000c10 tx=0x7f60d8009040 comp rx=0 tx=0).stop 2024-03-16T20:03:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 msgr2=0x7f60e414e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:03:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f60d00b0040 tx=0x7f60d0019810 comp rx=0 tx=0).stop 2024-03-16T20:03:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 shutdown_connections 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60c8066ec0 0x7f60c8069380 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e4143500 0x7f60e4142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60e41e2480 0x7f60e414e600 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 --2- 172.21.15.32:0/3717978119 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60e41e20a0 0x7f60e4144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 >> 172.21.15.32:0/3717978119 conn(0x7f60e406c0d0 msgr2=0x7f60e40c5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 shutdown_connections 2024-03-16T20:03:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:03:58.350+0000 7f60eb153640 1 -- 172.21.15.32:0/3717978119 wait complete. 2024-03-16T20:03:58.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:03:58.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:03:59.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:00.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 Processor -- start 2024-03-16T20:04:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 -- start start 2024-03-16T20:04:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 0x7fe350156e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35015b3e0 con 0x7fe350151100 2024-03-16T20:04:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35015a840 con 0x7fe350150d20 2024-03-16T20:04:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe358737640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3500c41f0 con 0x7fe3500c7990 2024-03-16T20:04:00.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe356cd4640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe350151100 0x7fe3500c2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48578/0 (socket says 172.21.15.32:48578) 2024-03-16T20:04:00.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe356cd4640 1 -- 172.21.15.32:0/2619874842 learned_addr learned my addr 172.21.15.32:0/2619874842 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:00.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.082+0000 7fe355cd2640 1 --2- 172.21.15.32:0/2619874842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 0x7fe350156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:00.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.086+0000 7fe3554d1640 1 -- 172.21.15.32:0/2619874842 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707581034 0 0) 0x7fe3480014a0 con 0x7fe350151100 2024-03-16T20:04:00.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.086+0000 7fe3554d1640 1 -- 172.21.15.32:0/2619874842 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c003530 con 0x7fe350151100 2024-03-16T20:04:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.086+0000 7fe355cd2640 1 -- 172.21.15.32:0/2619874842 >> v1:172.21.15.59:6789/0 conn(0x7fe350150d20 legacy=0x7fe35015acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.086+0000 7fe355cd2640 1 -- 172.21.15.32:0/2619874842 >> v1:172.21.15.32:6789/0 conn(0x7fe350151100 legacy=0x7fe3500c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.086+0000 7fe355cd2640 1 -- 172.21.15.32:0/2619874842 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350071280 con 0x7fe3500c7990 2024-03-16T20:04:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.090+0000 7fe355cd2640 1 --2- 172.21.15.32:0/2619874842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 0x7fe350156e40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe3400097f0 tx=0x7fe34001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea128b95a6cd9bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:00.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe3554d1640 1 -- 172.21.15.32:0/2619874842 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400a89d0 con 0x7fe3500c7990 2024-03-16T20:04:00.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe3554d1640 1 -- 172.21.15.32:0/2619874842 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400a8b30 con 0x7fe3500c7990 2024-03-16T20:04:00.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe3554d1640 1 -- 172.21.15.32:0/2619874842 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400b0440 con 0x7fe3500c7990 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- 172.21.15.32:0/2619874842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 msgr2=0x7fe350156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 --2- 172.21.15.32:0/2619874842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 0x7fe350156e40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe3400097f0 tx=0x7fe34001ecd0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- 172.21.15.32:0/2619874842 shutdown_connections 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 --2- 172.21.15.32:0/2619874842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe3500c7990 0x7fe350156e40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- 172.21.15.32:0/2619874842 >> 172.21.15.32:0/2619874842 conn(0x7fe3500cb470 msgr2=0x7fe3500cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- 172.21.15.32:0/2619874842 shutdown_connections 2024-03-16T20:04:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- 172.21.15.32:0/2619874842 wait complete. 2024-03-16T20:04:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 Processor -- start 2024-03-16T20:04:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.098+0000 7fe358737640 1 -- start start 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3500c7990 0x7fe35013fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe350151100 0x7fe35015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35015b580 con 0x7fe350150d20 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35013f690 con 0x7fe3500c7990 2024-03-16T20:04:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe358737640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35015b7b0 con 0x7fe350151100 2024-03-16T20:04:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe355cd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe355cd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35520/0 (socket says 172.21.15.32:35520) 2024-03-16T20:04:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe355cd2640 1 -- 172.21.15.32:0/3315754585 learned_addr learned my addr 172.21.15.32:0/3315754585 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe3564d3640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3500c7990 0x7fe35013fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.102+0000 7fe356cd4640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe350151100 0x7fe35015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.106+0000 7fe355cd2640 1 -- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe350151100 msgr2=0x7fe35015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.106+0000 7fe355cd2640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe350151100 0x7fe35015be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.106+0000 7fe355cd2640 1 -- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3500c7990 msgr2=0x7fe35013fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.106+0000 7fe355cd2640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3500c7990 0x7fe35013fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.106+0000 7fe355cd2640 1 -- 172.21.15.32:0/3315754585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340009420 con 0x7fe350150d20 2024-03-16T20:04:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.110+0000 7fe355cd2640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe34001b8e0 tx=0x7fe34001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400b9630 con 0x7fe350150d20 2024-03-16T20:04:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400b97b0 con 0x7fe350150d20 2024-03-16T20:04:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400b9b60 con 0x7fe350150d20 2024-03-16T20:04:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35013ed60 con 0x7fe350150d20 2024-03-16T20:04:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35013f320 con 0x7fe350150d20 2024-03-16T20:04:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.118+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35006a340 con 0x7fe350150d20 2024-03-16T20:04:00.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.154+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe3400b7040 con 0x7fe350150d20 2024-03-16T20:04:00.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.154+0000 7fe346ffd640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 0x7fe334069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:00.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.154+0000 7fe3564d3640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 0x7fe334069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:00.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.154+0000 7fe3564d3640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 0x7fe334069300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe34c001810 tx=0x7fe34c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:00.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.154+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe3400f4090 con 0x7fe350150d20 2024-03-16T20:04:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.438+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe3400f0330 con 0x7fe350150d20 2024-03-16T20:04:00.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.638+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35014e100 con 0x7fe350150d20 2024-03-16T20:04:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.682+0000 7fe346ffd640 1 -- 172.21.15.32:0/3315754585 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe3400b5030 con 0x7fe350150d20 2024-03-16T20:04:00.665 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:00.665 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 msgr2=0x7fe334069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 0x7fe334069300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe34c001810 tx=0x7fe34c009040 comp rx=0 tx=0).stop 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 msgr2=0x7fe350140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe34001b8e0 tx=0x7fe34001ba70 comp rx=0 tx=0).stop 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 shutdown_connections 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe334066e40 0x7fe334069300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe350151100 0x7fe35015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350150d20 0x7fe350140060 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 --2- 172.21.15.32:0/3315754585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3500c7990 0x7fe35013fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 >> 172.21.15.32:0/3315754585 conn(0x7fe3500cb470 msgr2=0x7fe35006f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 shutdown_connections 2024-03-16T20:04:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:00.686+0000 7fe358737640 1 -- 172.21.15.32:0/3315754585 wait complete. 2024-03-16T20:04:01.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:01.289 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:02.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:02.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.414+0000 7f8f32daf640 1 Processor -- start 2024-03-16T20:04:02.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.414+0000 7f8f32daf640 1 -- start start 2024-03-16T20:04:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f32daf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f32daf640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f2c072a10 0x7f8f2c072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f32daf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2c06bae0 con 0x7f8f2c154350 2024-03-16T20:04:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f32daf640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f2c15a540 con 0x7f8f2c072a10 2024-03-16T20:04:02.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f32daf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c0c7fd0 con 0x7f8f2c0c2d70 2024-03-16T20:04:02.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f3134c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f2c072a10 0x7f8f2c072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f30b4b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8f2c154350 0x7f8f2c15a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48582/0 (socket says 172.21.15.32:48582) 2024-03-16T20:04:02.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f30b4b640 1 -- 172.21.15.32:0/1434590408 learned_addr learned my addr 172.21.15.32:0/1434590408 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:02.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.418+0000 7f8f2bfff640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.422+0000 7f8f3134c640 1 -- 172.21.15.32:0/1434590408 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 msgr2=0x7f8f2c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:02.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.422+0000 7f8f3134c640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.422+0000 7f8f3134c640 1 -- 172.21.15.32:0/1434590408 >> v1:172.21.15.32:6789/0 conn(0x7f8f2c154350 legacy=0x7f8f2c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:02.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.422+0000 7f8f3134c640 1 -- 172.21.15.32:0/1434590408 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c06bd10 con 0x7f8f2c072a10 2024-03-16T20:04:02.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.426+0000 7f8f3134c640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c072a10 0x7f8f2c072e80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009d90 tx=0x7f8f2401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253960ba1f15cd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.430+0000 7f8f2b7fe640 1 -- 172.21.15.32:0/1434590408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240a57a0 con 0x7f8f2c072a10 2024-03-16T20:04:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f2b7fe640 1 -- 172.21.15.32:0/1434590408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f240a5900 con 0x7f8f2c072a10 2024-03-16T20:04:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f2b7fe640 1 -- 172.21.15.32:0/1434590408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240a5c90 con 0x7f8f2c072a10 2024-03-16T20:04:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 -- 172.21.15.32:0/1434590408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c072a10 msgr2=0x7f8f2c072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c072a10 0x7f8f2c072e80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009d90 tx=0x7f8f2401c8e0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 -- 172.21.15.32:0/1434590408 shutdown_connections 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c072a10 0x7f8f2c072e80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 --2- 172.21.15.32:0/1434590408 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 -- 172.21.15.32:0/1434590408 >> 172.21.15.32:0/1434590408 conn(0x7f8f2c06afc0 msgr2=0x7f8f2c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 -- 172.21.15.32:0/1434590408 shutdown_connections 2024-03-16T20:04:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.434+0000 7f8f32daf640 1 -- 172.21.15.32:0/1434590408 wait complete. 2024-03-16T20:04:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 Processor -- start 2024-03-16T20:04:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 -- start start 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f2c072a10 0x7f8f2c0c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c15b220 con 0x7f8f2c154350 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c0720d0 con 0x7f8f2c072a10 2024-03-16T20:04:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f32daf640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c072670 con 0x7f8f2c0c2d70 2024-03-16T20:04:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f3134c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f3134c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35540/0 (socket says 172.21.15.32:35540) 2024-03-16T20:04:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f3134c640 1 -- 172.21.15.32:0/4125557127 learned_addr learned my addr 172.21.15.32:0/4125557127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f2bfff640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.438+0000 7f8f30b4b640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f2c072a10 0x7f8f2c0c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.442+0000 7f8f3134c640 1 -- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 msgr2=0x7f8f2c155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.442+0000 7f8f3134c640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.442+0000 7f8f3134c640 1 -- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f2c072a10 msgr2=0x7f8f2c0c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.442+0000 7f8f3134c640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f2c072a10 0x7f8f2c0c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.442+0000 7f8f3134c640 1 -- 172.21.15.32:0/4125557127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f24009a70 con 0x7f8f2c154350 2024-03-16T20:04:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.446+0000 7f8f3134c640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8f2401af90 tx=0x7f8f24019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240b9650 con 0x7f8f2c154350 2024-03-16T20:04:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f240b97b0 con 0x7f8f2c154350 2024-03-16T20:04:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2c072390 con 0x7f8f2c154350 2024-03-16T20:04:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240b9b20 con 0x7f8f2c154350 2024-03-16T20:04:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2c1557c0 con 0x7f8f2c154350 2024-03-16T20:04:02.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.450+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2c0674e0 con 0x7f8f2c154350 2024-03-16T20:04:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.474+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8f240b9c80 con 0x7f8f2c154350 2024-03-16T20:04:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.478+0000 7f8f297fa640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 0x7f8f10069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.478+0000 7f8f30b4b640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 0x7f8f10069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.478+0000 7f8f30b4b640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 0x7f8f10069380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000f70 tx=0x7f8f20007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:02.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.478+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8f240f0bd0 con 0x7f8f2c154350 2024-03-16T20:04:02.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.730+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8f240ece70 con 0x7f8f2c154350 2024-03-16T20:04:02.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:02.994+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f2c1559b0 con 0x7f8f2c154350 2024-03-16T20:04:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.014+0000 7f8f297fa640 1 -- 172.21.15.32:0/4125557127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8f240f0530 con 0x7f8f2c154350 2024-03-16T20:04:02.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:02.999 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:03.000 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:03.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 msgr2=0x7f8f10069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 0x7f8f10069380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000f70 tx=0x7f8f20007e90 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 msgr2=0x7f8f2c1f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8f2401af90 tx=0x7f8f24019680 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 shutdown_connections 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8f10066ec0 0x7f8f10069380 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f2c154350 0x7f8f2c1f6450 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8f2c0c2d70 0x7f8f2c155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 --2- 172.21.15.32:0/4125557127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f2c072a10 0x7f8f2c0c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 >> 172.21.15.32:0/4125557127 conn(0x7f8f2c06afc0 msgr2=0x7f8f2c0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 shutdown_connections 2024-03-16T20:04:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:03.018+0000 7f8f32daf640 1 -- 172.21.15.32:0/4125557127 wait complete. 2024-03-16T20:04:03.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:03.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:04.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:04.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.746+0000 7f9b3b9c3640 1 Processor -- start 2024-03-16T20:04:04.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.746+0000 7f9b3b9c3640 1 -- start start 2024-03-16T20:04:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3b9c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b34151ef0 0x7f9b340c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3b9c3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3b9c3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b34158c80 con 0x7f9b340c8720 2024-03-16T20:04:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3b9c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b340b0ad0 con 0x7f9b34151ef0 2024-03-16T20:04:04.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3b9c3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34157cb0 con 0x7f9b340c97b0 2024-03-16T20:04:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b39f60640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9b340c8720 0x7f9b3415b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48592/0 (socket says 172.21.15.32:48592) 2024-03-16T20:04:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b39f60640 1 -- 172.21.15.32:0/3537109182 learned_addr learned my addr 172.21.15.32:0/3537109182 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/3537109182 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.750+0000 7f9b3975f640 1 --2- 172.21.15.32:0/3537109182 >> v2:172.21.15.59:3300/0 conn(0x7f9b34151ef0 0x7f9b340c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:04.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.754+0000 7f9b2bfff640 1 -- 172.21.15.32:0/3537109182 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607048346 0 0) 0x7f9b34158c80 con 0x7f9b340c8720 2024-03-16T20:04:04.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.754+0000 7f9b2bfff640 1 -- 172.21.15.32:0/3537109182 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b20003530 con 0x7f9b340c8720 2024-03-16T20:04:04.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.758+0000 7f9b38f5e640 1 -- 172.21.15.32:0/3537109182 >> v2:172.21.15.59:3300/0 conn(0x7f9b34151ef0 msgr2=0x7f9b340c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:04.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.758+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/3537109182 >> v2:172.21.15.59:3300/0 conn(0x7f9b34151ef0 0x7f9b340c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:04.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.758+0000 7f9b38f5e640 1 -- 172.21.15.32:0/3537109182 >> v1:172.21.15.32:6789/0 conn(0x7f9b340c8720 legacy=0x7f9b3415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:04.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.758+0000 7f9b38f5e640 1 -- 172.21.15.32:0/3537109182 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3415bfc0 con 0x7f9b340c97b0 2024-03-16T20:04:04.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.762+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/3537109182 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c81e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b240097c0 tx=0x7f9b2401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8932a4729698e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b2bfff640 1 -- 172.21.15.32:0/3537109182 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b240a8990 con 0x7f9b340c97b0 2024-03-16T20:04:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b2bfff640 1 -- 172.21.15.32:0/3537109182 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b240a8af0 con 0x7f9b340c97b0 2024-03-16T20:04:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b2bfff640 1 -- 172.21.15.32:0/3537109182 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b240a8e80 con 0x7f9b340c97b0 2024-03-16T20:04:04.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/3537109182 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 msgr2=0x7f9b340c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:04.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/3537109182 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c81e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b240097c0 tx=0x7f9b2401ec90 comp rx=0 tx=0).stop 2024-03-16T20:04:04.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/3537109182 shutdown_connections 2024-03-16T20:04:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/3537109182 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c81e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/3537109182 >> v2:172.21.15.59:3300/0 conn(0x7f9b34151ef0 0x7f9b340c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/3537109182 >> 172.21.15.32:0/3537109182 conn(0x7f9b34070f90 msgr2=0x7f9b340713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/3537109182 shutdown_connections 2024-03-16T20:04:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/3537109182 wait complete. 2024-03-16T20:04:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 Processor -- start 2024-03-16T20:04:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.774+0000 7f9b3b9c3640 1 -- start start 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b34151ef0 0x7f9b340c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340c61c0 con 0x7f9b340c8720 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340b1ec0 con 0x7f9b34151ef0 2024-03-16T20:04:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3b9c3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3414c2b0 con 0x7f9b340c97b0 2024-03-16T20:04:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3975f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3975f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35546/0 (socket says 172.21.15.32:35546) 2024-03-16T20:04:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b3975f640 1 -- 172.21.15.32:0/404346946 learned_addr learned my addr 172.21.15.32:0/404346946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.778+0000 7f9b39f60640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b34151ef0 0x7f9b340c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:04.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.782+0000 7f9b3975f640 1 -- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 msgr2=0x7f9b340c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:04.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.782+0000 7f9b3975f640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:04.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.782+0000 7f9b3975f640 1 -- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b34151ef0 msgr2=0x7f9b340c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:04.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.782+0000 7f9b3975f640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b34151ef0 0x7f9b340c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:04.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.782+0000 7f9b3975f640 1 -- 172.21.15.32:0/404346946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b24009420 con 0x7f9b340c8720 2024-03-16T20:04:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.790+0000 7f9b3975f640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000e870 tx=0x7f9b3000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30009510 con 0x7f9b340c8720 2024-03-16T20:04:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3000eec0 con 0x7f9b340c8720 2024-03-16T20:04:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b340b35e0 con 0x7f9b340c8720 2024-03-16T20:04:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3000cb40 con 0x7f9b340c8720 2024-03-16T20:04:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3415b030 con 0x7f9b340c8720 2024-03-16T20:04:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:04.794+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b340b1cb0 con 0x7f9b340c8720 2024-03-16T20:04:05.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.078+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9b3000cca0 con 0x7f9b340c8720 2024-03-16T20:04:05.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.078+0000 7f9b29ffb640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 0x7f9b18069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.082+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 0x7f9b18069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.082+0000 7f9b38f5e640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 0x7f9b18069300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9b240b2000 tx=0x7f9b24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.098+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9b30055ee0 con 0x7f9b340c8720 2024-03-16T20:04:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.098+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9b30052180 con 0x7f9b340c8720 2024-03-16T20:04:05.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.394+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b34068000 con 0x7f9b340c8720 2024-03-16T20:04:05.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.418+0000 7f9b29ffb640 1 -- 172.21.15.32:0/404346946 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9b30017090 con 0x7f9b340c8720 2024-03-16T20:04:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:05.402 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:05.402 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 msgr2=0x7f9b18069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 0x7f9b18069300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9b240b2000 tx=0x7f9b24008040 comp rx=0 tx=0).stop 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 msgr2=0x7f9b340b2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000e870 tx=0x7f9b3000ec70 comp rx=0 tx=0).stop 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 shutdown_connections 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b18066e40 0x7f9b18069300 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b34151ef0 0x7f9b340c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:05.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b340c97b0 0x7f9b340c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:05.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 --2- 172.21.15.32:0/404346946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340c8720 0x7f9b340b2350 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:05.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 >> 172.21.15.32:0/404346946 conn(0x7f9b34070f90 msgr2=0x7f9b340c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:05.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 shutdown_connections 2024-03-16T20:04:05.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:05.422+0000 7f9b3b9c3640 1 -- 172.21.15.32:0/404346946 wait complete. 2024-03-16T20:04:06.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:06.025 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:07.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:07.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.150+0000 7fc11b039640 1 Processor -- start 2024-03-16T20:04:07.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.150+0000 7fc11b039640 1 -- start start 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc1140b1eb0 0x7fc114072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1140c7ce0 0x7fc114072f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc114158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1140c68e0 con 0x7fc1140c7ce0 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc114067340 con 0x7fc1140b1eb0 2024-03-16T20:04:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc11b039640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11406c1f0 con 0x7fc11406ea80 2024-03-16T20:04:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1140c7ce0 0x7fc114072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1140c7ce0 0x7fc114072f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35550/0 (socket says 172.21.15.32:35550) 2024-03-16T20:04:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 -- 172.21.15.32:0/1599470398 learned_addr learned my addr 172.21.15.32:0/1599470398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc118dd5640 1 --2- 172.21.15.32:0/1599470398 >> v2:172.21.15.59:3300/0 conn(0x7fc1140b1eb0 0x7fc114072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc1195d6640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc114158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 -- 172.21.15.32:0/1599470398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 msgr2=0x7fc114158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:07.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc114158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 -- 172.21.15.32:0/1599470398 >> v2:172.21.15.59:3300/0 conn(0x7fc1140b1eb0 msgr2=0x7fc114072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:07.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 --2- 172.21.15.32:0/1599470398 >> v2:172.21.15.59:3300/0 conn(0x7fc1140b1eb0 0x7fc114072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.154+0000 7fc113fff640 1 -- 172.21.15.32:0/1599470398 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11406c4d0 con 0x7fc1140c7ce0 2024-03-16T20:04:07.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.158+0000 7fc113fff640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140c7ce0 0x7fc114072f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc114074a60 tx=0x7fc10001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db18fad91da02d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc1137fe640 1 -- 172.21.15.32:0/1599470398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000a89d0 con 0x7fc1140c7ce0 2024-03-16T20:04:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc1137fe640 1 -- 172.21.15.32:0/1599470398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1000a8b30 con 0x7fc1140c7ce0 2024-03-16T20:04:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc1137fe640 1 -- 172.21.15.32:0/1599470398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000b0440 con 0x7fc1140c7ce0 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- 172.21.15.32:0/1599470398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140c7ce0 msgr2=0x7fc114072f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140c7ce0 0x7fc114072f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc114074a60 tx=0x7fc10001ed10 comp rx=0 tx=0).stop 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- 172.21.15.32:0/1599470398 shutdown_connections 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc114158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- 172.21.15.32:0/1599470398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140c7ce0 0x7fc114072f50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- 172.21.15.32:0/1599470398 >> v2:172.21.15.59:3300/0 conn(0x7fc1140b1eb0 0x7fc114072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- 172.21.15.32:0/1599470398 >> 172.21.15.32:0/1599470398 conn(0x7fc11406afc0 msgr2=0x7fc11406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- 172.21.15.32:0/1599470398 shutdown_connections 2024-03-16T20:04:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- 172.21.15.32:0/1599470398 wait complete. 2024-03-16T20:04:07.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 Processor -- start 2024-03-16T20:04:07.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- start start 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc11414c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1140c7ce0 0x7fc114155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11415b480 con 0x7fc1140b1eb0 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11415ae30 con 0x7fc1140c7ce0 2024-03-16T20:04:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc11b039640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11415b110 con 0x7fc11406ea80 2024-03-16T20:04:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.166+0000 7fc113fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.170+0000 7fc113fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35558/0 (socket says 172.21.15.32:35558) 2024-03-16T20:04:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.170+0000 7fc113fff640 1 -- 172.21.15.32:0/3878916975 learned_addr learned my addr 172.21.15.32:0/3878916975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.170+0000 7fc1195d6640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1140c7ce0 0x7fc114155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.170+0000 7fc118dd5640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc11414c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.174+0000 7fc113fff640 1 -- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 msgr2=0x7fc11414c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.174+0000 7fc113fff640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc11414c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.174+0000 7fc113fff640 1 -- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1140c7ce0 msgr2=0x7fc114155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.174+0000 7fc113fff640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1140c7ce0 0x7fc114155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.174+0000 7fc113fff640 1 -- 172.21.15.32:0/3878916975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000094d0 con 0x7fc1140b1eb0 2024-03-16T20:04:07.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.178+0000 7fc113fff640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc1000b3040 tx=0x7fc10001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:07.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000b0d60 con 0x7fc1140b1eb0 2024-03-16T20:04:07.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1000bb460 con 0x7fc1140b1eb0 2024-03-16T20:04:07.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc114159f60 con 0x7fc1140b1eb0 2024-03-16T20:04:07.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000bb810 con 0x7fc1140b1eb0 2024-03-16T20:04:07.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11415a490 con 0x7fc1140b1eb0 2024-03-16T20:04:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.186+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11415af70 con 0x7fc1140b1eb0 2024-03-16T20:04:07.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.222+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc1000b83f0 con 0x7fc1140b1eb0 2024-03-16T20:04:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.222+0000 7fc1117fa640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 0x7fc0f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.222+0000 7fc118dd5640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 0x7fc0f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.222+0000 7fc118dd5640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 0x7fc0f8069300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc108009b60 tx=0x7fc108009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:07.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.230+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc1000f33c0 con 0x7fc1140b1eb0 2024-03-16T20:04:07.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.502+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc1000ef660 con 0x7fc1140b1eb0 2024-03-16T20:04:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.806+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11415af70 con 0x7fc1140b1eb0 2024-03-16T20:04:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.830+0000 7fc1117fa640 1 -- 172.21.15.32:0/3878916975 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc11415af70 con 0x7fc1140b1eb0 2024-03-16T20:04:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:07.814 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:07.814 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 msgr2=0x7fc0f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:07.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 0x7fc0f8069300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc108009b60 tx=0x7fc108009190 comp rx=0 tx=0).stop 2024-03-16T20:04:07.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 msgr2=0x7fc114154d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:07.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc1000b3040 tx=0x7fc10001bab0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 shutdown_connections 2024-03-16T20:04:07.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc0f8066e40 0x7fc0f8069300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1140c7ce0 0x7fc114155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140b1eb0 0x7fc114154d10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 --2- 172.21.15.32:0/3878916975 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc11406ea80 0x7fc11414c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 >> 172.21.15.32:0/3878916975 conn(0x7fc11406afc0 msgr2=0x7fc11406d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 shutdown_connections 2024-03-16T20:04:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:07.834+0000 7fc11b039640 1 -- 172.21.15.32:0/3878916975 wait complete. 2024-03-16T20:04:08.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:08.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:09.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 Processor -- start 2024-03-16T20:04:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 -- start start 2024-03-16T20:04:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f70c4151000 0x7f70c40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70c40c36b0 0x7f70c415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c40c8ec0 con 0x7f70c40c8980 2024-03-16T20:04:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c4072f10 con 0x7f70c40c36b0 2024-03-16T20:04:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70ca5f3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c40c5000 con 0x7f70c4151000 2024-03-16T20:04:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70c37fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70c40c36b0 0x7f70c415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70c3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f70c4151000 0x7f70c40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70c37fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70c40c36b0 0x7f70c415a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50992/0 (socket says 172.21.15.32:50992) 2024-03-16T20:04:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.574+0000 7f70c37fe640 1 -- 172.21.15.32:0/2434267766 learned_addr learned my addr 172.21.15.32:0/2434267766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.578+0000 7f70c37fe640 1 -- 172.21.15.32:0/2434267766 >> v2:172.21.15.59:3300/0 conn(0x7f70c4151000 msgr2=0x7f70c40c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.578+0000 7f70c37fe640 1 --2- 172.21.15.32:0/2434267766 >> v2:172.21.15.59:3300/0 conn(0x7f70c4151000 0x7f70c40c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.578+0000 7f70c37fe640 1 -- 172.21.15.32:0/2434267766 >> v1:172.21.15.59:6789/0 conn(0x7f70c40c8980 legacy=0x7f70c40c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.578+0000 7f70c37fe640 1 -- 172.21.15.32:0/2434267766 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4072950 con 0x7f70c40c36b0 2024-03-16T20:04:09.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.582+0000 7f70c37fe640 1 --2- 172.21.15.32:0/2434267766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40c36b0 0x7f70c415a240 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f70c406ef10 tx=0x7f70b801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee889f32dd1454f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.586+0000 7f70c2ffd640 1 -- 172.21.15.32:0/2434267766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80aa640 con 0x7f70c40c36b0 2024-03-16T20:04:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.586+0000 7f70c2ffd640 1 -- 172.21.15.32:0/2434267766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b80aa7a0 con 0x7f70c40c36b0 2024-03-16T20:04:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.586+0000 7f70c2ffd640 1 -- 172.21.15.32:0/2434267766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80aab30 con 0x7f70c40c36b0 2024-03-16T20:04:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2434267766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40c36b0 msgr2=0x7f70c415a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2434267766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40c36b0 0x7f70c415a240 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f70c406ef10 tx=0x7f70b801c4e0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2434267766 shutdown_connections 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2434267766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40c36b0 0x7f70c415a240 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2434267766 >> v2:172.21.15.59:3300/0 conn(0x7f70c4151000 0x7f70c40c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2434267766 >> 172.21.15.32:0/2434267766 conn(0x7f70c406c0d0 msgr2=0x7f70c406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2434267766 shutdown_connections 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2434267766 wait complete. 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 Processor -- start 2024-03-16T20:04:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- start start 2024-03-16T20:04:09.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70c40c36b0 0x7f70c4141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70c40c8980 0x7f70c415f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4160c00 con 0x7f70c4151000 2024-03-16T20:04:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4156220 con 0x7f70c40c36b0 2024-03-16T20:04:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70ca5f3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4160d70 con 0x7f70c40c8980 2024-03-16T20:04:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70c8b90640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70c8b90640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51000/0 (socket says 172.21.15.32:51000) 2024-03-16T20:04:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.590+0000 7f70c8b90640 1 -- 172.21.15.32:0/2274054702 learned_addr learned my addr 172.21.15.32:0/2274054702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.594+0000 7f70c3fff640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70c40c36b0 0x7f70c4141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.594+0000 7f70c37fe640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70c40c8980 0x7f70c415f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.598+0000 7f70c8b90640 1 -- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70c40c8980 msgr2=0x7f70c415f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.598+0000 7f70c8b90640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70c40c8980 0x7f70c415f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.598+0000 7f70c8b90640 1 -- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70c40c36b0 msgr2=0x7f70c4141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.598+0000 7f70c8b90640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70c40c36b0 0x7f70c4141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:09.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.598+0000 7f70c8b90640 1 -- 172.21.15.32:0/2274054702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80094d0 con 0x7f70c4151000 2024-03-16T20:04:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.602+0000 7f70c8b90640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00c780 tx=0x7f70bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc003d90 con 0x7f70c4151000 2024-03-16T20:04:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c41612f0 con 0x7f70c4151000 2024-03-16T20:04:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c41f2fc0 con 0x7f70c4151000 2024-03-16T20:04:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c4068000 con 0x7f70c4151000 2024-03-16T20:04:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc00ce90 con 0x7f70c4151000 2024-03-16T20:04:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.606+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc006d20 con 0x7f70c4151000 2024-03-16T20:04:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.922+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f70bc005070 con 0x7f70c4151000 2024-03-16T20:04:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.922+0000 7f70c0ff9640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 0x7f70b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.922+0000 7f70c3fff640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 0x7f70b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.922+0000 7f70c3fff640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 0x7f70b0069300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009ac0 tx=0x7f70b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.938+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f70bc0552c0 con 0x7f70c4151000 2024-03-16T20:04:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:09.938+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f70bc051560 con 0x7f70c4151000 2024-03-16T20:04:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.170+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70c4073410 con 0x7f70c4151000 2024-03-16T20:04:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70c0ff9640 1 -- 172.21.15.32:0/2274054702 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f70bc017090 con 0x7f70c4151000 2024-03-16T20:04:10.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:10.177 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 msgr2=0x7f70b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 0x7f70b0069300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009ac0 tx=0x7f70b4009190 comp rx=0 tx=0).stop 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 msgr2=0x7f70c414f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00c780 tx=0x7f70bc00cc40 comp rx=0 tx=0).stop 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 shutdown_connections 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f70b0066e40 0x7f70b0069300 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c4151000 0x7f70c414f740 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70c40c8980 0x7f70c415f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 --2- 172.21.15.32:0/2274054702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70c40c36b0 0x7f70c4141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.194+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 >> 172.21.15.32:0/2274054702 conn(0x7f70c406c0d0 msgr2=0x7f70c406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.198+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 shutdown_connections 2024-03-16T20:04:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:10.198+0000 7f70ca5f3640 1 -- 172.21.15.32:0/2274054702 wait complete. 2024-03-16T20:04:10.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:10.799 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:11.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 Processor -- start 2024-03-16T20:04:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 -- start start 2024-03-16T20:04:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32180c6040 0x7f32180c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f32180c6420 0x7f3218157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3218070760 con 0x7f321806ba30 2024-03-16T20:04:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32180731c0 con 0x7f32180c6420 2024-03-16T20:04:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.926+0000 7f321f176640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32180734a0 con 0x7f32180c6040 2024-03-16T20:04:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f32180c6420 0x7f3218157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f32180c6420 0x7f3218157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51014/0 (socket says 172.21.15.32:51014) 2024-03-16T20:04:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 -- 172.21.15.32:0/2184450839 learned_addr learned my addr 172.21.15.32:0/2184450839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f321cf12640 1 --2- 172.21.15.32:0/2184450839 >> v2:172.21.15.59:3300/0 conn(0x7f32180c6040 0x7f32180c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 -- 172.21.15.32:0/2184450839 >> v2:172.21.15.59:3300/0 conn(0x7f32180c6040 msgr2=0x7f32180c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:11.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 --2- 172.21.15.32:0/2184450839 >> v2:172.21.15.59:3300/0 conn(0x7f32180c6040 0x7f32180c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:11.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 -- 172.21.15.32:0/2184450839 >> v1:172.21.15.32:6789/0 conn(0x7f321806ba30 legacy=0x7f32181516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:11.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.930+0000 7f3217fff640 1 -- 172.21.15.32:0/2184450839 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321806fe30 con 0x7f32180c6420 2024-03-16T20:04:11.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.938+0000 7f3217fff640 1 --2- 172.21.15.32:0/2184450839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6420 0x7f3218157c80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f32180c3aa0 tx=0x7f320401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef351fb754bd3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:11.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f32177fe640 1 -- 172.21.15.32:0/2184450839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040ab600 con 0x7f32180c6420 2024-03-16T20:04:11.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f32177fe640 1 -- 172.21.15.32:0/2184450839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32040a7050 con 0x7f32180c6420 2024-03-16T20:04:11.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f32177fe640 1 -- 172.21.15.32:0/2184450839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040ab980 con 0x7f32180c6420 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 -- 172.21.15.32:0/2184450839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6420 msgr2=0x7f3218157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 --2- 172.21.15.32:0/2184450839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6420 0x7f3218157c80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f32180c3aa0 tx=0x7f320401c420 comp rx=0 tx=0).stop 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 -- 172.21.15.32:0/2184450839 shutdown_connections 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 --2- 172.21.15.32:0/2184450839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6420 0x7f3218157c80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 --2- 172.21.15.32:0/2184450839 >> v2:172.21.15.59:3300/0 conn(0x7f32180c6040 0x7f32180c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 -- 172.21.15.32:0/2184450839 >> 172.21.15.32:0/2184450839 conn(0x7f321806c0d0 msgr2=0x7f321806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 -- 172.21.15.32:0/2184450839 shutdown_connections 2024-03-16T20:04:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.942+0000 7f321f176640 1 -- 172.21.15.32:0/2184450839 wait complete. 2024-03-16T20:04:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 Processor -- start 2024-03-16T20:04:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 -- start start 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321806ba30 0x7f32180caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32180c6420 0x7f32181f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218162250 con 0x7f32180c6040 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32180c4f50 con 0x7f321806ba30 2024-03-16T20:04:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321f176640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32181f6970 con 0x7f32180c6420 2024-03-16T20:04:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f3217fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f3217fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-03-16T20:04:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f3217fff640 1 -- 172.21.15.32:0/1000195740 learned_addr learned my addr 172.21.15.32:0/1000195740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321d713640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32180c6420 0x7f32181f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.946+0000 7f321cf12640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321806ba30 0x7f32180caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.950+0000 7f3217fff640 1 -- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32180c6420 msgr2=0x7f32181f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.950+0000 7f3217fff640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32180c6420 0x7f32181f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.950+0000 7f3217fff640 1 -- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321806ba30 msgr2=0x7f32180caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.950+0000 7f3217fff640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321806ba30 0x7f32180caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.950+0000 7f3217fff640 1 -- 172.21.15.32:0/1000195740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040094d0 con 0x7f32180c6040 2024-03-16T20:04:11.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.954+0000 7f3217fff640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f32040ae040 tx=0x7f32040191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040acea0 con 0x7f32180c6040 2024-03-16T20:04:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32040b6640 con 0x7f32180c6040 2024-03-16T20:04:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040b69f0 con 0x7f32180c6040 2024-03-16T20:04:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f321f176640 1 -- 172.21.15.32:0/1000195740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32181f6c10 con 0x7f32180c6040 2024-03-16T20:04:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f321f176640 1 -- 172.21.15.32:0/1000195740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32181f71d0 con 0x7f32180c6040 2024-03-16T20:04:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.958+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00052a0 con 0x7f32180c6040 2024-03-16T20:04:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.986+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f32040b2040 con 0x7f32180c6040 2024-03-16T20:04:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.986+0000 7f32157fa640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 0x7f31fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.986+0000 7f321cf12640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 0x7f31fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.986+0000 7f321cf12640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 0x7f31fc069300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f320c007a60 tx=0x7f320c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:11.990+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f32040f0290 con 0x7f32180c6040 2024-03-16T20:04:12.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.282+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f32040ec530 con 0x7f32180c6040 2024-03-16T20:04:12.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:04:12.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:04:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.506+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f0005900 con 0x7f32180c6040 2024-03-16T20:04:12.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.526+0000 7f32157fa640 1 -- 172.21.15.32:0/1000195740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f31f0005900 con 0x7f32180c6040 2024-03-16T20:04:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:12.510 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 msgr2=0x7f31fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 0x7f31fc069300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f320c007a60 tx=0x7f320c008040 comp rx=0 tx=0).stop 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 msgr2=0x7f32181f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f32040ae040 tx=0x7f32040191c0 comp rx=0 tx=0).stop 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 shutdown_connections 2024-03-16T20:04:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f31fc066e40 0x7f31fc069300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32180c6420 0x7f32181f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32180c6040 0x7f32181f33f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 --2- 172.21.15.32:0/1000195740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321806ba30 0x7f32180caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 >> 172.21.15.32:0/1000195740 conn(0x7f321806c0d0 msgr2=0x7f32180ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 shutdown_connections 2024-03-16T20:04:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:12.530+0000 7f3202ffd640 1 -- 172.21.15.32:0/1000195740 wait complete. 2024-03-16T20:04:13.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:13.134 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:14.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:14.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.258+0000 7f85392e7640 1 Processor -- start 2024-03-16T20:04:14.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.258+0000 7f85392e7640 1 -- start start 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.258+0000 7f85392e7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85340c6040 0x7f85340c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.258+0000 7f85392e7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85340c6420 0x7f8534157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85392e7640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534070760 con 0x7f853406ba30 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85392e7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340731c0 con 0x7f85340c6420 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85392e7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340734a0 con 0x7f85340c6040 2024-03-16T20:04:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85327fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85340c6420 0x7f8534157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85327fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85340c6420 0x7f8534157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51030/0 (socket says 172.21.15.32:51030) 2024-03-16T20:04:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f85327fc640 1 -- 172.21.15.32:0/276537914 learned_addr learned my addr 172.21.15.32:0/276537914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.262+0000 7f8532ffd640 1 --2- 172.21.15.32:0/276537914 >> v2:172.21.15.59:3300/0 conn(0x7f85340c6040 0x7f85340c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.266+0000 7f85327fc640 1 -- 172.21.15.32:0/276537914 >> v2:172.21.15.59:3300/0 conn(0x7f85340c6040 msgr2=0x7f85340c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.266+0000 7f85327fc640 1 --2- 172.21.15.32:0/276537914 >> v2:172.21.15.59:3300/0 conn(0x7f85340c6040 0x7f85340c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.266+0000 7f85327fc640 1 -- 172.21.15.32:0/276537914 >> v1:172.21.15.32:6789/0 conn(0x7f853406ba30 legacy=0x7f85341516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.266+0000 7f85327fc640 1 -- 172.21.15.32:0/276537914 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853406fe30 con 0x7f85340c6420 2024-03-16T20:04:14.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.274+0000 7f85327fc640 1 --2- 172.21.15.32:0/276537914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85340c6420 0x7f8534157c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f85340c3aa0 tx=0x7f852801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19e52dfdc2f5822 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f8531ffb640 1 -- 172.21.15.32:0/276537914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280ab600 con 0x7f85340c6420 2024-03-16T20:04:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f8531ffb640 1 -- 172.21.15.32:0/276537914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85280a7050 con 0x7f85340c6420 2024-03-16T20:04:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f8531ffb640 1 -- 172.21.15.32:0/276537914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280ab980 con 0x7f85340c6420 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 -- 172.21.15.32:0/276537914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85340c6420 msgr2=0x7f8534157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 --2- 172.21.15.32:0/276537914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85340c6420 0x7f8534157c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f85340c3aa0 tx=0x7f852801c460 comp rx=0 tx=0).stop 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 -- 172.21.15.32:0/276537914 shutdown_connections 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 --2- 172.21.15.32:0/276537914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85340c6420 0x7f8534157c80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 --2- 172.21.15.32:0/276537914 >> v2:172.21.15.59:3300/0 conn(0x7f85340c6040 0x7f85340c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 -- 172.21.15.32:0/276537914 >> 172.21.15.32:0/276537914 conn(0x7f853406c0d0 msgr2=0x7f853406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 -- 172.21.15.32:0/276537914 shutdown_connections 2024-03-16T20:04:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.278+0000 7f85392e7640 1 -- 172.21.15.32:0/276537914 wait complete. 2024-03-16T20:04:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 Processor -- start 2024-03-16T20:04:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 -- start start 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85340c6040 0x7f85341435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85340c6420 0x7f853413f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85341f31a0 con 0x7f853406ba30 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85341494f0 con 0x7f85340c6420 2024-03-16T20:04:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85392e7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85340c5660 con 0x7f85340c6040 2024-03-16T20:04:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f8532ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85340c6420 0x7f853413f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f85327fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85340c6040 0x7f85341435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f8532ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51040/0 (socket says 172.21.15.32:51040) 2024-03-16T20:04:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.282+0000 7f8532ffd640 1 -- 172.21.15.32:0/1780448161 learned_addr learned my addr 172.21.15.32:0/1780448161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.290+0000 7f8532ffd640 1 -- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85340c6040 msgr2=0x7f85341435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.290+0000 7f8532ffd640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85340c6040 0x7f85341435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.290+0000 7f8532ffd640 1 -- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85340c6420 msgr2=0x7f853413f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.290+0000 7f8532ffd640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85340c6420 0x7f853413f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.290+0000 7f8532ffd640 1 -- 172.21.15.32:0/1780448161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280094d0 con 0x7f853406ba30 2024-03-16T20:04:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.294+0000 7f8532ffd640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f852400e9e0 tx=0x7f852400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85240096b0 con 0x7f853406ba30 2024-03-16T20:04:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852400c9d0 con 0x7f853406ba30 2024-03-16T20:04:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f85392e7640 1 -- 172.21.15.32:0/1780448161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8534149ee0 con 0x7f853406ba30 2024-03-16T20:04:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852400cd60 con 0x7f853406ba30 2024-03-16T20:04:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f85392e7640 1 -- 172.21.15.32:0/1780448161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85341f2ec0 con 0x7f853406ba30 2024-03-16T20:04:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.298+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0052a0 con 0x7f853406ba30 2024-03-16T20:04:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.334+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8524013470 con 0x7f853406ba30 2024-03-16T20:04:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.334+0000 7f851b7fe640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 0x7f8520069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.334+0000 7f85327fc640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 0x7f8520069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:14.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.334+0000 7f85327fc640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 0x7f8520069300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f85280ad000 tx=0x7f85280a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.522+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8524056090 con 0x7f853406ba30 2024-03-16T20:04:14.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.626+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8524052330 con 0x7f853406ba30 2024-03-16T20:04:14.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.842+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c005900 con 0x7f853406ba30 2024-03-16T20:04:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.858+0000 7f851b7fe640 1 -- 172.21.15.32:0/1780448161 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f850c005900 con 0x7f853406ba30 2024-03-16T20:04:14.843 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:14.843 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 msgr2=0x7f8520069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 0x7f8520069300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f85280ad000 tx=0x7f85280a6000 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 msgr2=0x7f85340c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f852400e9e0 tx=0x7f852400eea0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 shutdown_connections 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8520066e40 0x7f8520069300 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85340c6420 0x7f853413f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85340c6040 0x7f85341435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 --2- 172.21.15.32:0/1780448161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f853406ba30 0x7f85340c9300 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 >> 172.21.15.32:0/1780448161 conn(0x7f853406c0d0 msgr2=0x7f8534150c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 shutdown_connections 2024-03-16T20:04:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:14.862+0000 7f8519ffb640 1 -- 172.21.15.32:0/1780448161 wait complete. 2024-03-16T20:04:15.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:15.468 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:16.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 Processor -- start 2024-03-16T20:04:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 -- start start 2024-03-16T20:04:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9ac0714f0 0x7fd9ac0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0ca1d0 con 0x7fd9ac071110 2024-03-16T20:04:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0c34a0 con 0x7fd9ac0c2910 2024-03-16T20:04:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.594+0000 7fd9b0af5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0c9ba0 con 0x7fd9ac0714f0 2024-03-16T20:04:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9ac0714f0 0x7fd9ac0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9ac0714f0 0x7fd9ac0ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51046/0 (socket says 172.21.15.32:51046) 2024-03-16T20:04:16.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/2735545071 learned_addr learned my addr 172.21.15.32:0/2735545071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/2735545071 >> v1:172.21.15.59:6789/0 conn(0x7fd9ac0c2910 legacy=0x7fd9ac151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/2735545071 >> v1:172.21.15.32:6789/0 conn(0x7fd9ac071110 legacy=0x7fd9ac0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.598+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/2735545071 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0c2ee0 con 0x7fd9ac0714f0 2024-03-16T20:04:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.606+0000 7fd9a9d9b640 1 --2- 172.21.15.32:0/2735545071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac0ca740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009870 tx=0x7fd9a001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ec6a904ee13816 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9a959a640 1 -- 172.21.15.32:0/2735545071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00a8a10 con 0x7fd9ac0714f0 2024-03-16T20:04:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9a959a640 1 -- 172.21.15.32:0/2735545071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a00a8b70 con 0x7fd9ac0714f0 2024-03-16T20:04:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9a959a640 1 -- 172.21.15.32:0/2735545071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00b0440 con 0x7fd9ac0714f0 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 -- 172.21.15.32:0/2735545071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 msgr2=0x7fd9ac0ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/2735545071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac0ca740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009870 tx=0x7fd9a001ed10 comp rx=0 tx=0).stop 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 -- 172.21.15.32:0/2735545071 shutdown_connections 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/2735545071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac0ca740 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 -- 172.21.15.32:0/2735545071 >> 172.21.15.32:0/2735545071 conn(0x7fd9ac06afc0 msgr2=0x7fd9ac06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 -- 172.21.15.32:0/2735545071 shutdown_connections 2024-03-16T20:04:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.610+0000 7fd9b0af5640 1 -- 172.21.15.32:0/2735545071 wait complete. 2024-03-16T20:04:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 Processor -- start 2024-03-16T20:04:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 -- start start 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac071110 0x7fd9ac06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9ac0c2910 0x7fd9ac1ef970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac158670 con 0x7fd9ac0714f0 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac06dff0 con 0x7fd9ac071110 2024-03-16T20:04:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.614+0000 7fd9b0af5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac1f2ee0 con 0x7fd9ac0c2910 2024-03-16T20:04:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.618+0000 7fd9a9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.618+0000 7fd9a9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51052/0 (socket says 172.21.15.32:51052) 2024-03-16T20:04:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.618+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/3044938996 learned_addr learned my addr 172.21.15.32:0/3044938996 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.618+0000 7fd9aad9d640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9ac0c2910 0x7fd9ac1ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.618+0000 7fd9aa59c640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac071110 0x7fd9ac06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.622+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9ac0c2910 msgr2=0x7fd9ac1ef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.622+0000 7fd9a9d9b640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9ac0c2910 0x7fd9ac1ef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.622+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac071110 msgr2=0x7fd9ac06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.622+0000 7fd9a9d9b640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac071110 0x7fd9ac06d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.622+0000 7fd9a9d9b640 1 -- 172.21.15.32:0/3044938996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a00094d0 con 0x7fd9ac0714f0 2024-03-16T20:04:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.626+0000 7fd9a9d9b640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00b3040 tx=0x7fd9a001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:16.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.630+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00bf070 con 0x7fd9ac0714f0 2024-03-16T20:04:16.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.630+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a00b0c80 con 0x7fd9ac0714f0 2024-03-16T20:04:16.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.630+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac0b2e20 con 0x7fd9ac0714f0 2024-03-16T20:04:16.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.630+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00b9410 con 0x7fd9ac0714f0 2024-03-16T20:04:16.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.630+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac1f34a0 con 0x7fd9ac0714f0 2024-03-16T20:04:16.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.634+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac068000 con 0x7fd9ac0714f0 2024-03-16T20:04:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.658+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd9a00b7040 con 0x7fd9ac0714f0 2024-03-16T20:04:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.658+0000 7fd99affd640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 0x7fd98c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.658+0000 7fd9aa59c640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 0x7fd98c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.658+0000 7fd9aa59c640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 0x7fd98c069300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd99c003d60 tx=0x7fd99c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:16.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.946+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd9a00cb330 con 0x7fd9ac0714f0 2024-03-16T20:04:16.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:16.946+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd9a00ef780 con 0x7fd9ac0714f0 2024-03-16T20:04:17.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.150+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ac14f080 con 0x7fd9ac0714f0 2024-03-16T20:04:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.182+0000 7fd99affd640 1 -- 172.21.15.32:0/3044938996 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fd9a00b5020 con 0x7fd9ac0714f0 2024-03-16T20:04:17.165 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:17.166 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:03:17.448123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:04:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 msgr2=0x7fd98c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 0x7fd98c069300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd99c003d60 tx=0x7fd99c009040 comp rx=0 tx=0).stop 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 msgr2=0x7fd9ac06c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00b3040 tx=0x7fd9a001bab0 comp rx=0 tx=0).stop 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 shutdown_connections 2024-03-16T20:04:17.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd98c066e40 0x7fd98c069300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9ac0c2910 0x7fd9ac1ef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9ac0714f0 0x7fd9ac06c930 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 --2- 172.21.15.32:0/3044938996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac071110 0x7fd9ac06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 >> 172.21.15.32:0/3044938996 conn(0x7fd9ac06afc0 msgr2=0x7fd9ac0ca6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 shutdown_connections 2024-03-16T20:04:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:17.186+0000 7fd9b0af5640 1 -- 172.21.15.32:0/3044938996 wait complete. 2024-03-16T20:04:17.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:03:17.448123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:04:17.791 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:18.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.918+0000 7f40d906e640 1 Processor -- start 2024-03-16T20:04:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.918+0000 7f40d906e640 1 -- start start 2024-03-16T20:04:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d906e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 0x7f40d40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d906e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d906e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d4158c80 con 0x7f40d40c8720 2024-03-16T20:04:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d906e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d40b0ad0 con 0x7f40d4151ef0 2024-03-16T20:04:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d906e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4157cb0 con 0x7f40d40c97b0 2024-03-16T20:04:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 0x7f40d40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 0x7f40d40c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:42126/0 (socket says 172.21.15.32:42126) 2024-03-16T20:04:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.922+0000 7f40d2d9d640 1 -- 172.21.15.32:0/2464889487 learned_addr learned my addr 172.21.15.32:0/2464889487 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.930+0000 7f40d259c640 1 -- 172.21.15.32:0/2464889487 >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 msgr2=0x7f40d40c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.930+0000 7f40d259c640 1 --2- 172.21.15.32:0/2464889487 >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 0x7f40d40c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.930+0000 7f40d259c640 1 -- 172.21.15.32:0/2464889487 >> v1:172.21.15.59:6789/0 conn(0x7f40d40c8720 legacy=0x7f40d415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:18.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.930+0000 7f40d259c640 1 -- 172.21.15.32:0/2464889487 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d415bfc0 con 0x7f40d40c97b0 2024-03-16T20:04:18.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.934+0000 7f40d259c640 1 --2- 172.21.15.32:0/2464889487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c81e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40c80097c0 tx=0x7f40c801ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b54f0ffbde0b5a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d1d9b640 1 -- 172.21.15.32:0/2464889487 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c80a8910 con 0x7f40d40c97b0 2024-03-16T20:04:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d1d9b640 1 -- 172.21.15.32:0/2464889487 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c80a8a70 con 0x7f40d40c97b0 2024-03-16T20:04:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d1d9b640 1 -- 172.21.15.32:0/2464889487 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c80a8e00 con 0x7f40d40c97b0 2024-03-16T20:04:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- 172.21.15.32:0/2464889487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 msgr2=0x7f40d40c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 --2- 172.21.15.32:0/2464889487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c81e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40c80097c0 tx=0x7f40c801ec50 comp rx=0 tx=0).stop 2024-03-16T20:04:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- 172.21.15.32:0/2464889487 shutdown_connections 2024-03-16T20:04:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 --2- 172.21.15.32:0/2464889487 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c81e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 --2- 172.21.15.32:0/2464889487 >> v2:172.21.15.59:3300/0 conn(0x7f40d4151ef0 0x7f40d40c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- 172.21.15.32:0/2464889487 >> 172.21.15.32:0/2464889487 conn(0x7f40d4070f90 msgr2=0x7f40d40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- 172.21.15.32:0/2464889487 shutdown_connections 2024-03-16T20:04:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- 172.21.15.32:0/2464889487 wait complete. 2024-03-16T20:04:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 Processor -- start 2024-03-16T20:04:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.942+0000 7f40d906e640 1 -- start start 2024-03-16T20:04:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40d4151ef0 0x7f40d406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40b0750 con 0x7f40d40c8720 2024-03-16T20:04:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d406caa0 con 0x7f40d4151ef0 2024-03-16T20:04:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d906e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d406da30 con 0x7f40d40c97b0 2024-03-16T20:04:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51068/0 (socket says 172.21.15.32:51068) 2024-03-16T20:04:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d2d9d640 1 -- 172.21.15.32:0/3113324133 learned_addr learned my addr 172.21.15.32:0/3113324133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.946+0000 7f40d359e640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40d4151ef0 0x7f40d406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.950+0000 7f40d2d9d640 1 -- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 msgr2=0x7f40d40c3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.950+0000 7f40d2d9d640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.950+0000 7f40d2d9d640 1 -- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40d4151ef0 msgr2=0x7f40d406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.950+0000 7f40d2d9d640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40d4151ef0 0x7f40d406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.950+0000 7f40d2d9d640 1 -- 172.21.15.32:0/3113324133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c8009420 con 0x7f40d40c8720 2024-03-16T20:04:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.954+0000 7f40d2d9d640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40c400e820 tx=0x7f40c400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.958+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4009580 con 0x7f40d40c8720 2024-03-16T20:04:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.958+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c400c940 con 0x7f40d40c8720 2024-03-16T20:04:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.958+0000 7f40d906e640 1 -- 172.21.15.32:0/3113324133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d40b0190 con 0x7f40d40c8720 2024-03-16T20:04:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.958+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c400ccf0 con 0x7f40d40c8720 2024-03-16T20:04:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.962+0000 7f40d906e640 1 -- 172.21.15.32:0/3113324133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d406d6c0 con 0x7f40d40c8720 2024-03-16T20:04:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.962+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac0052a0 con 0x7f40d40c8720 2024-03-16T20:04:18.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.998+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f40c4013470 con 0x7f40d40c8720 2024-03-16T20:04:18.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.998+0000 7f40bb7fe640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 0x7f40c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:18.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.998+0000 7f40d259c640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 0x7f40c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:18.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:18.998+0000 7f40d259c640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 0x7f40c0069300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b2000 tx=0x7f40c80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.294+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f40c4055e60 con 0x7f40d40c8720 2024-03-16T20:04:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.294+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f40c4052100 con 0x7f40d40c8720 2024-03-16T20:04:19.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.538+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40ac005900 con 0x7f40d40c8720 2024-03-16T20:04:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.562+0000 7f40bb7fe640 1 -- 172.21.15.32:0/3113324133 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f40ac005900 con 0x7f40d40c8720 2024-03-16T20:04:19.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:19.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:19.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 msgr2=0x7f40c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:19.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 0x7f40c0069300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b2000 tx=0x7f40c80aa070 comp rx=0 tx=0).stop 2024-03-16T20:04:19.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 msgr2=0x7f40d41f2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:19.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40c400e820 tx=0x7f40c400ece0 comp rx=0 tx=0).stop 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 shutdown_connections 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f40c0066e40 0x7f40c0069300 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40d4151ef0 0x7f40d406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40d40c97b0 0x7f40d40c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 --2- 172.21.15.32:0/3113324133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d40c8720 0x7f40d41f2ee0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 >> 172.21.15.32:0/3113324133 conn(0x7f40d4070f90 msgr2=0x7f40d4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 shutdown_connections 2024-03-16T20:04:19.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:19.566+0000 7f40b97fa640 1 -- 172.21.15.32:0/3113324133 wait complete. 2024-03-16T20:04:20.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:20.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:21.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 Processor -- start 2024-03-16T20:04:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 -- start start 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5f8153fa0 0x7ff5f8070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff5f80c7470 0x7ff5f815d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f815d920 0x7ff5f80724a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f80c3f00 con 0x7ff5f8153fa0 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f8070650 con 0x7ff5f80c7470 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5fceb9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80c8590 con 0x7ff5f815d920 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5f659c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5f8153fa0 0x7ff5f8070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5effff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff5f80c7470 0x7ff5f815d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5f659c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5f8153fa0 0x7ff5f8070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38746/0 (socket says 172.21.15.32:38746) 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5f659c640 1 -- 172.21.15.32:0/2629237304 learned_addr learned my addr 172.21.15.32:0/2629237304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.334+0000 7ff5f6d9d640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f815d920 0x7ff5f80724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.338+0000 7ff5f659c640 1 -- 172.21.15.32:0/2629237304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f815d920 msgr2=0x7ff5f80724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:21.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.338+0000 7ff5f659c640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f815d920 0x7ff5f80724a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.338+0000 7ff5f659c640 1 -- 172.21.15.32:0/2629237304 >> v2:172.21.15.59:3300/0 conn(0x7ff5f80c7470 msgr2=0x7ff5f815d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:21.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.338+0000 7ff5f659c640 1 --2- 172.21.15.32:0/2629237304 >> v2:172.21.15.59:3300/0 conn(0x7ff5f80c7470 0x7ff5f815d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.338+0000 7ff5f659c640 1 -- 172.21.15.32:0/2629237304 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8073590 con 0x7ff5f8153fa0 2024-03-16T20:04:21.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.342+0000 7ff5f659c640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8070840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff5f806d0b0 tx=0x7ff5e801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb26497f50f86b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:21.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5f5d9b640 1 -- 172.21.15.32:0/2629237304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80a89d0 con 0x7ff5f8153fa0 2024-03-16T20:04:21.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5f5d9b640 1 -- 172.21.15.32:0/2629237304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e80a8b30 con 0x7ff5f8153fa0 2024-03-16T20:04:21.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5f5d9b640 1 -- 172.21.15.32:0/2629237304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80b0440 con 0x7ff5f8153fa0 2024-03-16T20:04:21.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 -- 172.21.15.32:0/2629237304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 msgr2=0x7ff5f8070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8070840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff5f806d0b0 tx=0x7ff5e801ed10 comp rx=0 tx=0).stop 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 -- 172.21.15.32:0/2629237304 shutdown_connections 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f815d920 0x7ff5f80724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/2629237304 >> v2:172.21.15.59:3300/0 conn(0x7ff5f80c7470 0x7ff5f815d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/2629237304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8070840 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 -- 172.21.15.32:0/2629237304 >> 172.21.15.32:0/2629237304 conn(0x7ff5f806c0d0 msgr2=0x7ff5f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 -- 172.21.15.32:0/2629237304 shutdown_connections 2024-03-16T20:04:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.346+0000 7ff5fceb9640 1 -- 172.21.15.32:0/2629237304 wait complete. 2024-03-16T20:04:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 Processor -- start 2024-03-16T20:04:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 -- start start 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f80c7470 0x7ff5f81f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f815d920 0x7ff5f814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f81efb50 con 0x7ff5f8153fa0 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f81486f0 con 0x7ff5f815d920 2024-03-16T20:04:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5fceb9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80ca760 con 0x7ff5f80c7470 2024-03-16T20:04:21.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5effff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5f659c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f80c7470 0x7ff5f81f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5f6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f815d920 0x7ff5f814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5effff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38760/0 (socket says 172.21.15.32:38760) 2024-03-16T20:04:21.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.350+0000 7ff5effff640 1 -- 172.21.15.32:0/1942317326 learned_addr learned my addr 172.21.15.32:0/1942317326 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:21.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.354+0000 7ff5effff640 1 -- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f80c7470 msgr2=0x7ff5f81f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:21.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.354+0000 7ff5effff640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f80c7470 0x7ff5f81f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.354+0000 7ff5effff640 1 -- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f815d920 msgr2=0x7ff5f814f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:21.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.354+0000 7ff5effff640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f815d920 0x7ff5f814f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.354+0000 7ff5effff640 1 -- 172.21.15.32:0/1942317326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e80094d0 con 0x7ff5f8153fa0 2024-03-16T20:04:21.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.358+0000 7ff5effff640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000e8b0 tx=0x7ff5e000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.362+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000c940 con 0x7ff5f8153fa0 2024-03-16T20:04:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.362+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e000caa0 con 0x7ff5f8153fa0 2024-03-16T20:04:21.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.362+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000ce30 con 0x7ff5f8153fa0 2024-03-16T20:04:21.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.362+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8148920 con 0x7ff5f8153fa0 2024-03-16T20:04:21.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.362+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f81efd40 con 0x7ff5f8153fa0 2024-03-16T20:04:21.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.366+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8153b10 con 0x7ff5f8153fa0 2024-03-16T20:04:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.390+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff5e0013470 con 0x7ff5f8153fa0 2024-03-16T20:04:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.390+0000 7ff5eeffd640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 0x7ff5d8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.390+0000 7ff5f659c640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 0x7ff5d8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.390+0000 7ff5f659c640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 0x7ff5d8069350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80b2000 tx=0x7ff5e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.394+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff5e0055eb0 con 0x7ff5f8153fa0 2024-03-16T20:04:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.662+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff5e0052150 con 0x7ff5f8153fa0 2024-03-16T20:04:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.858+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f80ca130 con 0x7ff5f8153fa0 2024-03-16T20:04:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.870+0000 7ff5eeffd640 1 -- 172.21.15.32:0/1942317326 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff5e0021070 con 0x7ff5f8153fa0 2024-03-16T20:04:21.854 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:21.854 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 msgr2=0x7ff5d8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 0x7ff5d8069350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80b2000 tx=0x7ff5e8009040 comp rx=0 tx=0).stop 2024-03-16T20:04:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 msgr2=0x7ff5f8155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000e8b0 tx=0x7ff5e000ed70 comp rx=0 tx=0).stop 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 shutdown_connections 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff5d8066e90 0x7ff5d8069350 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f815d920 0x7ff5f814f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f8153fa0 0x7ff5f8155f90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 --2- 172.21.15.32:0/1942317326 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5f80c7470 0x7ff5f81f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 >> 172.21.15.32:0/1942317326 conn(0x7ff5f806c0d0 msgr2=0x7ff5f80c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 shutdown_connections 2024-03-16T20:04:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:21.874+0000 7ff5fceb9640 1 -- 172.21.15.32:0/1942317326 wait complete. 2024-03-16T20:04:22.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:22.484 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:23.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.610+0000 7f93d94dd640 1 Processor -- start 2024-03-16T20:04:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.610+0000 7f93d94dd640 1 -- start start 2024-03-16T20:04:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d94dd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93d4072c70 0x7f93d40c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d94dd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4151330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d94dd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d406e6a0 con 0x7f93d4073050 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d94dd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d4156f20 con 0x7f93d4072c70 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d94dd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d406c250 con 0x7f93d406bae0 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d2ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93d4072c70 0x7f93d40c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d27fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f93d4073050 0x7f93d4150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:38582/0 (socket says 172.21.15.32:38582) 2024-03-16T20:04:23.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d27fc640 1 -- 172.21.15.32:0/2810757758 learned_addr learned my addr 172.21.15.32:0/2810757758 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.614+0000 7f93d37fe640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:23.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.618+0000 7f93d2ffd640 1 -- 172.21.15.32:0/2810757758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 msgr2=0x7f93d4151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:23.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.618+0000 7f93d2ffd640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:23.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.618+0000 7f93d2ffd640 1 -- 172.21.15.32:0/2810757758 >> v1:172.21.15.59:6789/0 conn(0x7f93d4073050 legacy=0x7f93d4150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:23.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.618+0000 7f93d2ffd640 1 -- 172.21.15.32:0/2810757758 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d406c510 con 0x7f93d4072c70 2024-03-16T20:04:23.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.622+0000 7f93d2ffd640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4072c70 0x7f93d40c0f90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f93c40097c0 tx=0x7f93c401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb7851494269942 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:23.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d1ffb640 1 -- 172.21.15.32:0/2810757758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c40aa5f0 con 0x7f93d4072c70 2024-03-16T20:04:23.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d1ffb640 1 -- 172.21.15.32:0/2810757758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c40aa750 con 0x7f93d4072c70 2024-03-16T20:04:23.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d1ffb640 1 -- 172.21.15.32:0/2810757758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c40aaae0 con 0x7f93d4072c70 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 -- 172.21.15.32:0/2810757758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4072c70 msgr2=0x7f93d40c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4072c70 0x7f93d40c0f90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f93c40097c0 tx=0x7f93c401c470 comp rx=0 tx=0).stop 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 -- 172.21.15.32:0/2810757758 shutdown_connections 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 --2- 172.21.15.32:0/2810757758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4072c70 0x7f93d40c0f90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 -- 172.21.15.32:0/2810757758 >> 172.21.15.32:0/2810757758 conn(0x7f93d406afc0 msgr2=0x7f93d406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 -- 172.21.15.32:0/2810757758 shutdown_connections 2024-03-16T20:04:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.626+0000 7f93d94dd640 1 -- 172.21.15.32:0/2810757758 wait complete. 2024-03-16T20:04:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 Processor -- start 2024-03-16T20:04:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 -- start start 2024-03-16T20:04:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4073050 0x7f93d41f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d41404a0 con 0x7f93d4073050 2024-03-16T20:04:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40c5150 con 0x7f93d4072c70 2024-03-16T20:04:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d94dd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4141900 con 0x7f93d406bae0 2024-03-16T20:04:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.630+0000 7f93d2ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.634+0000 7f93d27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:59500/0 (socket says 172.21.15.32:59500) 2024-03-16T20:04:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.634+0000 7f93d27fc640 1 -- 172.21.15.32:0/1695923707 learned_addr learned my addr 172.21.15.32:0/1695923707 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.638+0000 7f93d27fc640 1 -- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 msgr2=0x7f93d4142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.638+0000 7f93d27fc640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.638+0000 7f93d27fc640 1 -- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4073050 msgr2=0x7f93d41f0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.638+0000 7f93d27fc640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4073050 0x7f93d41f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.638+0000 7f93d27fc640 1 -- 172.21.15.32:0/1695923707 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4009420 con 0x7f93d4072c70 2024-03-16T20:04:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.642+0000 7f93d27fc640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e740 tx=0x7f93c800ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:23.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.646+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8005cf0 con 0x7f93d4072c70 2024-03-16T20:04:23.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.646+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8005e50 con 0x7f93d4072c70 2024-03-16T20:04:23.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.646+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d41436f0 con 0x7f93d4072c70 2024-03-16T20:04:23.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.646+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80072f0 con 0x7f93d4072c70 2024-03-16T20:04:23.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.646+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d4145840 con 0x7f93d4072c70 2024-03-16T20:04:23.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.650+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4145a60 con 0x7f93d4072c70 2024-03-16T20:04:23.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.938+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f93c8007450 con 0x7f93d4072c70 2024-03-16T20:04:23.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.938+0000 7f93bb7fe640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 0x7f93c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:23.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.938+0000 7f93d2ffd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 0x7f93c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:23.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.938+0000 7f93d2ffd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 0x7f93c0069350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f93c40ae000 tx=0x7f93c40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:23.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.946+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f93c8057cb0 con 0x7f93d4072c70 2024-03-16T20:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:23.962+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f93c8053f50 con 0x7f93d4072c70 2024-03-16T20:04:24.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.170+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93d4068000 con 0x7f93d4072c70 2024-03-16T20:04:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.194+0000 7f93bb7fe640 1 -- 172.21.15.32:0/1695923707 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f93c801f090 con 0x7f93d4072c70 2024-03-16T20:04:24.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:24.177 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:24.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 msgr2=0x7f93c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 0x7f93c0069350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f93c40ae000 tx=0x7f93c40a7000 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 msgr2=0x7f93d40c4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e740 tx=0x7f93c800ec00 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 shutdown_connections 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f93c0066e90 0x7f93c0069350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93d4073050 0x7f93d41f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d4072c70 0x7f93d40c4590 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 --2- 172.21.15.32:0/1695923707 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f93d406bae0 0x7f93d4142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 >> 172.21.15.32:0/1695923707 conn(0x7f93d406afc0 msgr2=0x7f93d4155cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 shutdown_connections 2024-03-16T20:04:24.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:24.198+0000 7f93d94dd640 1 -- 172.21.15.32:0/1695923707 wait complete. 2024-03-16T20:04:24.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:24.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:25.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 Processor -- start 2024-03-16T20:04:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 -- start start 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0b815ab30 0x7ff0b8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b8072dd0 con 0x7ff0b8150d70 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b815d790 con 0x7ff0b80725e0 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0bd763640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b815da50 con 0x7ff0b815ab30 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0b6ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0b8150d70 0x7ff0b8070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57922/0 (socket says 172.21.15.32:57922) 2024-03-16T20:04:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.926+0000 7ff0b6ffd640 1 -- 172.21.15.32:0/1081582874 learned_addr learned my addr 172.21.15.32:0/1081582874 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.930+0000 7ff0b77fe640 1 --2- 172.21.15.32:0/1081582874 >> v2:172.21.15.59:3300/0 conn(0x7ff0b815ab30 0x7ff0b8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.930+0000 7ff0b5ffb640 1 -- 172.21.15.32:0/1081582874 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580024819 0 0) 0x7ff0b8072dd0 con 0x7ff0b8150d70 2024-03-16T20:04:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.930+0000 7ff0b5ffb640 1 -- 172.21.15.32:0/1081582874 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a0003530 con 0x7ff0b8150d70 2024-03-16T20:04:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.934+0000 7ff0b77fe640 1 -- 172.21.15.32:0/1081582874 >> v1:172.21.15.59:6789/0 conn(0x7ff0b80725e0 legacy=0x7ff0b815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.934+0000 7ff0b77fe640 1 -- 172.21.15.32:0/1081582874 >> v1:172.21.15.32:6789/0 conn(0x7ff0b8150d70 legacy=0x7ff0b8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.934+0000 7ff0b77fe640 1 -- 172.21.15.32:0/1081582874 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b815dd30 con 0x7ff0b815ab30 2024-03-16T20:04:25.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.942+0000 7ff0b77fe640 1 --2- 172.21.15.32:0/1081582874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b815ab30 0x7ff0b8073070 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009d90 tx=0x7ff0b001c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aff23bc7d0a7964 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0b5ffb640 1 -- 172.21.15.32:0/1081582874 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00a58a0 con 0x7ff0b815ab30 2024-03-16T20:04:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0b5ffb640 1 -- 172.21.15.32:0/1081582874 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00a5a00 con 0x7ff0b815ab30 2024-03-16T20:04:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0b5ffb640 1 -- 172.21.15.32:0/1081582874 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00a5d90 con 0x7ff0b815ab30 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 -- 172.21.15.32:0/1081582874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b815ab30 msgr2=0x7ff0b8073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 --2- 172.21.15.32:0/1081582874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b815ab30 0x7ff0b8073070 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009d90 tx=0x7ff0b001c7e0 comp rx=0 tx=0).stop 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 -- 172.21.15.32:0/1081582874 shutdown_connections 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 --2- 172.21.15.32:0/1081582874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b815ab30 0x7ff0b8073070 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 -- 172.21.15.32:0/1081582874 >> 172.21.15.32:0/1081582874 conn(0x7ff0b806c0d0 msgr2=0x7ff0b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 -- 172.21.15.32:0/1081582874 shutdown_connections 2024-03-16T20:04:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.950+0000 7ff0bd763640 1 -- 172.21.15.32:0/1081582874 wait complete. 2024-03-16T20:04:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 Processor -- start 2024-03-16T20:04:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 -- start start 2024-03-16T20:04:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b80725e0 0x7ff0b80cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0b8150d70 0x7ff0b81483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b814ca90 con 0x7ff0b815ab30 2024-03-16T20:04:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b80cd570 con 0x7ff0b80725e0 2024-03-16T20:04:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0bd763640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81f6450 con 0x7ff0b8150d70 2024-03-16T20:04:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0b77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0b77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38792/0 (socket says 172.21.15.32:38792) 2024-03-16T20:04:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0b77fe640 1 -- 172.21.15.32:0/3078088839 learned_addr learned my addr 172.21.15.32:0/3078088839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.954+0000 7ff0b67fc640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0b8150d70 0x7ff0b81483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b6ffd640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b80725e0 0x7ff0b80cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b77fe640 1 -- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0b8150d70 msgr2=0x7ff0b81483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b77fe640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0b8150d70 0x7ff0b81483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b77fe640 1 -- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b80725e0 msgr2=0x7ff0b80cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b77fe640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b80725e0 0x7ff0b80cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.958+0000 7ff0b77fe640 1 -- 172.21.15.32:0/3078088839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0009a70 con 0x7ff0b815ab30 2024-03-16T20:04:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.962+0000 7ff0b77fe640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0b001a410 tx=0x7ff0b0019580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.966+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00b9860 con 0x7ff0b815ab30 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.966+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00b99c0 con 0x7ff0b815ab30 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.966+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b815d560 con 0x7ff0b815ab30 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.966+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81f6a10 con 0x7ff0b815ab30 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.966+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00b9d50 con 0x7ff0b815ab30 2024-03-16T20:04:25.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:25.970+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0980052a0 con 0x7ff0b815ab30 2024-03-16T20:04:26.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.266+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0b00ad440 con 0x7ff0b815ab30 2024-03-16T20:04:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.266+0000 7ff09f7fe640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 0x7ff0a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.266+0000 7ff0b6ffd640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 0x7ff0a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.266+0000 7ff0b6ffd640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 0x7ff0a4069300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8007fd0 tx=0x7ff0a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:26.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.294+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff0b00a5820 con 0x7ff0b815ab30 2024-03-16T20:04:26.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.294+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff0b00ecfe0 con 0x7ff0b815ab30 2024-03-16T20:04:26.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.486+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff098005900 con 0x7ff0b815ab30 2024-03-16T20:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff09f7fe640 1 -- 172.21.15.32:0/3078088839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff098005900 con 0x7ff0b815ab30 2024-03-16T20:04:26.496 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:26.496 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 msgr2=0x7ff0a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 0x7ff0a4069300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8007fd0 tx=0x7ff0a8009040 comp rx=0 tx=0).stop 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 msgr2=0x7ff0b8148900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0b001a410 tx=0x7ff0b0019580 comp rx=0 tx=0).stop 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 shutdown_connections 2024-03-16T20:04:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0a4066e40 0x7ff0a4069300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0b815ab30 0x7ff0b8148900 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0b8150d70 0x7ff0b81483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 --2- 172.21.15.32:0/3078088839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b80725e0 0x7ff0b80cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 >> 172.21.15.32:0/3078088839 conn(0x7ff0b806c0d0 msgr2=0x7ff0b8073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 shutdown_connections 2024-03-16T20:04:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:26.514+0000 7ff0bd763640 1 -- 172.21.15.32:0/3078088839 wait complete. 2024-03-16T20:04:27.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:27.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:28.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:28.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 Processor -- start 2024-03-16T20:04:28.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 -- start start 2024-03-16T20:04:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd3e0153fa0 0x7fd3e0070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e0072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00c8590 con 0x7fd3e00c7470 2024-03-16T20:04:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e00c3f00 con 0x7fd3e0153fa0 2024-03-16T20:04:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e6f60640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0070650 con 0x7fd3e00724a0 2024-03-16T20:04:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e4cfc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd3e0153fa0 0x7fd3e0070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3dffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd3e00c7470 0x7fd3e015d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57930/0 (socket says 172.21.15.32:57930) 2024-03-16T20:04:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3dffff640 1 -- 172.21.15.32:0/3100684510 learned_addr learned my addr 172.21.15.32:0/3100684510 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.250+0000 7fd3e54fd640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e0072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.254+0000 7fd3e4cfc640 1 -- 172.21.15.32:0/3100684510 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 msgr2=0x7fd3e0072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.254+0000 7fd3e4cfc640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e0072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.254+0000 7fd3e4cfc640 1 -- 172.21.15.32:0/3100684510 >> v1:172.21.15.32:6789/0 conn(0x7fd3e00c7470 legacy=0x7fd3e015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.254+0000 7fd3e4cfc640 1 -- 172.21.15.32:0/3100684510 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0073590 con 0x7fd3e0153fa0 2024-03-16T20:04:28.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.262+0000 7fd3e4cfc640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0070840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40097c0 tx=0x7fd3d401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de51ffbc39b4e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:28.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.266+0000 7fd3df7fe640 1 -- 172.21.15.32:0/3100684510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d40a8990 con 0x7fd3e0153fa0 2024-03-16T20:04:28.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.266+0000 7fd3df7fe640 1 -- 172.21.15.32:0/3100684510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d40a8af0 con 0x7fd3e0153fa0 2024-03-16T20:04:28.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.266+0000 7fd3df7fe640 1 -- 172.21.15.32:0/3100684510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d40a8e80 con 0x7fd3e0153fa0 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- 172.21.15.32:0/3100684510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 msgr2=0x7fd3e0070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0070840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40097c0 tx=0x7fd3d401ec90 comp rx=0 tx=0).stop 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- 172.21.15.32:0/3100684510 shutdown_connections 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e0072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/3100684510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0070840 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- 172.21.15.32:0/3100684510 >> 172.21.15.32:0/3100684510 conn(0x7fd3e006c0d0 msgr2=0x7fd3e006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- 172.21.15.32:0/3100684510 shutdown_connections 2024-03-16T20:04:28.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- 172.21.15.32:0/3100684510 wait complete. 2024-03-16T20:04:28.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 Processor -- start 2024-03-16T20:04:28.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- start start 2024-03-16T20:04:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e00cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e00c7470 0x7fd3e00ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0155650 con 0x7fd3e0153fa0 2024-03-16T20:04:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0159250 con 0x7fd3e00c7470 2024-03-16T20:04:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.270+0000 7fd3e6f60640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e013daf0 con 0x7fd3e00724a0 2024-03-16T20:04:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.274+0000 7fd3e54fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.274+0000 7fd3dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e00c7470 0x7fd3e00ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.274+0000 7fd3e54fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38814/0 (socket says 172.21.15.32:38814) 2024-03-16T20:04:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.274+0000 7fd3e54fd640 1 -- 172.21.15.32:0/244842282 learned_addr learned my addr 172.21.15.32:0/244842282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:28.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.274+0000 7fd3e4cfc640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e00cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 -- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 msgr2=0x7fd3e00cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e00cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 -- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e00c7470 msgr2=0x7fd3e00ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e00c7470 0x7fd3e00ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 -- 172.21.15.32:0/244842282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d4009420 con 0x7fd3e0153fa0 2024-03-16T20:04:28.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.278+0000 7fd3e54fd640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800efc0 tx=0x7fd3d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800ced0 con 0x7fd3e0153fa0 2024-03-16T20:04:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d800f040 con 0x7fd3e0153fa0 2024-03-16T20:04:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80146c0 con 0x7fd3e0153fa0 2024-03-16T20:04:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e01552e0 con 0x7fd3e0153fa0 2024-03-16T20:04:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e0159730 con 0x7fd3e0153fa0 2024-03-16T20:04:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.286+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e0155790 con 0x7fd3e0153fa0 2024-03-16T20:04:28.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.594+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd3d8014820 con 0x7fd3e0153fa0 2024-03-16T20:04:28.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.594+0000 7fd3dd7fa640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 0x7fd3c4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:28.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.594+0000 7fd3e4cfc640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 0x7fd3c4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:28.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.594+0000 7fd3e4cfc640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 0x7fd3c4069350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40b2000 tx=0x7fd3d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:28.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.606+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd3d8056680 con 0x7fd3e0153fa0 2024-03-16T20:04:28.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.606+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd3d8052920 con 0x7fd3e0153fa0 2024-03-16T20:04:28.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.818+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e0155790 con 0x7fd3e0153fa0 2024-03-16T20:04:28.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.842+0000 7fd3dd7fa640 1 -- 172.21.15.32:0/244842282 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fd3e0155790 con 0x7fd3e0153fa0 2024-03-16T20:04:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:28.827 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:28.827 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:28.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 msgr2=0x7fd3c4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 0x7fd3c4069350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40b2000 tx=0x7fd3d4008040 comp rx=0 tx=0).stop 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 msgr2=0x7fd3e0155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800efc0 tx=0x7fd3d800c490 comp rx=0 tx=0).stop 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 shutdown_connections 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd3c4066e90 0x7fd3c4069350 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3e0153fa0 0x7fd3e0155a70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e00c7470 0x7fd3e00ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 --2- 172.21.15.32:0/244842282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd3e00724a0 0x7fd3e00cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 >> 172.21.15.32:0/244842282 conn(0x7fd3e006c0d0 msgr2=0x7fd3e006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 shutdown_connections 2024-03-16T20:04:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:28.846+0000 7fd3e6f60640 1 -- 172.21.15.32:0/244842282 wait complete. 2024-03-16T20:04:29.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:29.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:30.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 Processor -- start 2024-03-16T20:04:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 -- start start 2024-03-16T20:04:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 0x7f6ce40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce40c8590 con 0x7f6ce4153fa0 2024-03-16T20:04:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce4073590 con 0x7f6ce40724a0 2024-03-16T20:04:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.582+0000 7f6ce9622640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40c3f00 con 0x7f6ce4070840 2024-03-16T20:04:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.586+0000 7f6ce2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6ce4153fa0 0x7f6ce415d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50308/0 (socket says 172.21.15.32:50308) 2024-03-16T20:04:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.586+0000 7f6ce27fc640 1 --2- 172.21.15.32:0/1603047657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 0x7f6ce40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.586+0000 7f6ce2ffd640 1 -- 172.21.15.32:0/1603047657 learned_addr learned my addr 172.21.15.32:0/1603047657 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.590+0000 7f6ce1ffb640 1 -- 172.21.15.32:0/1603047657 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885105718 0 0) 0x7f6ce40c8590 con 0x7f6ce4153fa0 2024-03-16T20:04:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.590+0000 7f6ce1ffb640 1 -- 172.21.15.32:0/1603047657 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ccc003530 con 0x7f6ce4153fa0 2024-03-16T20:04:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.590+0000 7f6ce27fc640 1 -- 172.21.15.32:0/1603047657 >> v1:172.21.15.59:6789/0 conn(0x7f6ce40724a0 legacy=0x7f6ce4164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.590+0000 7f6ce27fc640 1 -- 172.21.15.32:0/1603047657 >> v1:172.21.15.32:6789/0 conn(0x7f6ce4153fa0 legacy=0x7f6ce415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.590+0000 7f6ce27fc640 1 -- 172.21.15.32:0/1603047657 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4070c60 con 0x7f6ce4070840 2024-03-16T20:04:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.594+0000 7f6ce27fc640 1 --2- 172.21.15.32:0/1603047657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 0x7f6ce40c7470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80097f0 tx=0x7f6cd801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da527614ce465181 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce1ffb640 1 -- 172.21.15.32:0/1603047657 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80a89d0 con 0x7f6ce4070840 2024-03-16T20:04:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce1ffb640 1 -- 172.21.15.32:0/1603047657 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd80a8b30 con 0x7f6ce4070840 2024-03-16T20:04:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce1ffb640 1 -- 172.21.15.32:0/1603047657 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80b0440 con 0x7f6ce4070840 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 -- 172.21.15.32:0/1603047657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 msgr2=0x7f6ce40c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 --2- 172.21.15.32:0/1603047657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 0x7f6ce40c7470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80097f0 tx=0x7f6cd801ecd0 comp rx=0 tx=0).stop 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 -- 172.21.15.32:0/1603047657 shutdown_connections 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 --2- 172.21.15.32:0/1603047657 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce4070840 0x7f6ce40c7470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 -- 172.21.15.32:0/1603047657 >> 172.21.15.32:0/1603047657 conn(0x7f6ce406c0d0 msgr2=0x7f6ce406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 -- 172.21.15.32:0/1603047657 shutdown_connections 2024-03-16T20:04:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.598+0000 7f6ce9622640 1 -- 172.21.15.32:0/1603047657 wait complete. 2024-03-16T20:04:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 Processor -- start 2024-03-16T20:04:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 -- start start 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce4070840 0x7f6ce41f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce40724a0 0x7f6ce41f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40cbac0 con 0x7f6ce4153fa0 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40cb590 con 0x7f6ce4070840 2024-03-16T20:04:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce9622640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce41f30f0 con 0x7f6ce40724a0 2024-03-16T20:04:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41780/0 (socket says 172.21.15.32:41780) 2024-03-16T20:04:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce37fe640 1 -- 172.21.15.32:0/1105585787 learned_addr learned my addr 172.21.15.32:0/1105585787 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce2ffd640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce4070840 0x7f6ce41f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.602+0000 7f6ce27fc640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce40724a0 0x7f6ce41f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 -- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce40724a0 msgr2=0x7f6ce41f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce40724a0 0x7f6ce41f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 -- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce4070840 msgr2=0x7f6ce41f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce4070840 0x7f6ce41f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 -- 172.21.15.32:0/1105585787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd8009420 con 0x7f6ce4153fa0 2024-03-16T20:04:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.610+0000 7f6ce37fe640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00c740 tx=0x7f6cdc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc006d60 con 0x7f6ce4153fa0 2024-03-16T20:04:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc00ce50 con 0x7f6ce4153fa0 2024-03-16T20:04:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6ce9622640 1 -- 172.21.15.32:0/1105585787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce40cb7e0 con 0x7f6ce4153fa0 2024-03-16T20:04:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc003bc0 con 0x7f6ce4153fa0 2024-03-16T20:04:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6ce9622640 1 -- 172.21.15.32:0/1105585787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce4153780 con 0x7f6ce4153fa0 2024-03-16T20:04:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.618+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc0052a0 con 0x7f6ce4153fa0 2024-03-16T20:04:30.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.654+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6cdc003d20 con 0x7f6ce4153fa0 2024-03-16T20:04:30.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.654+0000 7f6ccb7fe640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 0x7f6cd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:30.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.654+0000 7f6ce2ffd640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 0x7f6cd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:30.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.654+0000 7f6ce2ffd640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 0x7f6cd0069300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4007fd0 tx=0x7f6cd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:30.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.966+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6cdc055e50 con 0x7f6ce4153fa0 2024-03-16T20:04:30.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:30.966+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6cdc0520f0 con 0x7f6ce4153fa0 2024-03-16T20:04:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.294+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cbc005900 con 0x7f6ce4153fa0 2024-03-16T20:04:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.346+0000 7f6ccb7fe640 1 -- 172.21.15.32:0/1105585787 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f6cbc005900 con 0x7f6ce4153fa0 2024-03-16T20:04:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:31.329 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 msgr2=0x7f6cd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 0x7f6cd0069300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4007fd0 tx=0x7f6cd4009040 comp rx=0 tx=0).stop 2024-03-16T20:04:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 msgr2=0x7f6ce41423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00c740 tx=0x7f6cdc00cc00 comp rx=0 tx=0).stop 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 shutdown_connections 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6cd0066e40 0x7f6cd0069300 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ce4153fa0 0x7f6ce41423c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6ce40724a0 0x7f6ce41f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 --2- 172.21.15.32:0/1105585787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce4070840 0x7f6ce41f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 >> 172.21.15.32:0/1105585787 conn(0x7f6ce406c0d0 msgr2=0x7f6ce4072a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 shutdown_connections 2024-03-16T20:04:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:31.350+0000 7f6cc97fa640 1 -- 172.21.15.32:0/1105585787 wait complete. 2024-03-16T20:04:31.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:31.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:32.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 Processor -- start 2024-03-16T20:04:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 -- start start 2024-03-16T20:04:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2410150d70 0x7f2410073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f241015a420 0x7f241015a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2410072dd0 con 0x7f24100725e0 2024-03-16T20:04:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241015d790 con 0x7f2410150d70 2024-03-16T20:04:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.082+0000 7f241769f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241015da50 con 0x7f241015a420 2024-03-16T20:04:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f2415c3c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f241015a420 0x7f241015a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2410150d70 0x7f2410073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2410150d70 0x7f2410073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41790/0 (socket says 172.21.15.32:41790) 2024-03-16T20:04:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 -- 172.21.15.32:0/2252459507 learned_addr learned my addr 172.21.15.32:0/2252459507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 -- 172.21.15.32:0/2252459507 >> v2:172.21.15.59:3300/0 conn(0x7f241015a420 msgr2=0x7f241015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 --2- 172.21.15.32:0/2252459507 >> v2:172.21.15.59:3300/0 conn(0x7f241015a420 0x7f241015a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 -- 172.21.15.32:0/2252459507 >> v1:172.21.15.59:6789/0 conn(0x7f24100725e0 legacy=0x7f2410070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.086+0000 7f241543b640 1 -- 172.21.15.32:0/2252459507 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241015dd30 con 0x7f2410150d70 2024-03-16T20:04:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.090+0000 7f241543b640 1 --2- 172.21.15.32:0/2252459507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2410150d70 0x7f2410073070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f24100c4a30 tx=0x7f240c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4052d8757cb7b965 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f2407fff640 1 -- 172.21.15.32:0/2252459507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0ad640 con 0x7f2410150d70 2024-03-16T20:04:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f2407fff640 1 -- 172.21.15.32:0/2252459507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c0ad7a0 con 0x7f2410150d70 2024-03-16T20:04:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f2407fff640 1 -- 172.21.15.32:0/2252459507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0adb30 con 0x7f2410150d70 2024-03-16T20:04:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 -- 172.21.15.32:0/2252459507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2410150d70 msgr2=0x7f2410073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 --2- 172.21.15.32:0/2252459507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2410150d70 0x7f2410073070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f24100c4a30 tx=0x7f240c01c390 comp rx=0 tx=0).stop 2024-03-16T20:04:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 -- 172.21.15.32:0/2252459507 shutdown_connections 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 --2- 172.21.15.32:0/2252459507 >> v2:172.21.15.59:3300/0 conn(0x7f241015a420 0x7f241015a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 --2- 172.21.15.32:0/2252459507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2410150d70 0x7f2410073070 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 -- 172.21.15.32:0/2252459507 >> 172.21.15.32:0/2252459507 conn(0x7f241006c0d0 msgr2=0x7f241006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 -- 172.21.15.32:0/2252459507 shutdown_connections 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.094+0000 7f241769f640 1 -- 172.21.15.32:0/2252459507 wait complete. 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 Processor -- start 2024-03-16T20:04:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 -- start start 2024-03-16T20:04:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2410150d70 0x7f2410152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f241015a420 0x7f24101530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24101526e0 con 0x7f24100725e0 2024-03-16T20:04:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241015bfe0 con 0x7f2410150d70 2024-03-16T20:04:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241769f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241015ca90 con 0x7f241015a420 2024-03-16T20:04:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241543b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f2414c3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2410150d70 0x7f2410152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241543b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41806/0 (socket says 172.21.15.32:41806) 2024-03-16T20:04:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.098+0000 7f241543b640 1 -- 172.21.15.32:0/2009310430 learned_addr learned my addr 172.21.15.32:0/2009310430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:33.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.106+0000 7f241543b640 1 -- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f241015a420 msgr2=0x7f24101530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:33.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.106+0000 7f241543b640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f241015a420 0x7f24101530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.106+0000 7f241543b640 1 -- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2410150d70 msgr2=0x7f2410152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:33.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.106+0000 7f241543b640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2410150d70 0x7f2410152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.106+0000 7f241543b640 1 -- 172.21.15.32:0/2009310430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0094d0 con 0x7f24100725e0 2024-03-16T20:04:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.110+0000 7f241543b640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f240c0b0040 tx=0x7f240c019130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:33.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.114+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0b8680 con 0x7f24100725e0 2024-03-16T20:04:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.114+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c0a6040 con 0x7f24100725e0 2024-03-16T20:04:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.114+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241015d560 con 0x7f24100725e0 2024-03-16T20:04:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.118+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0b8a20 con 0x7f24100725e0 2024-03-16T20:04:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.118+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241015cd30 con 0x7f24100725e0 2024-03-16T20:04:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.118+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2410068000 con 0x7f24100725e0 2024-03-16T20:04:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.502+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f240c0b8b80 con 0x7f24100725e0 2024-03-16T20:04:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.502+0000 7f2405ffb640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 0x7f23f4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.502+0000 7f2414c3a640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 0x7f23f4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:33.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.502+0000 7f2414c3a640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 0x7f23f4069380 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24101f6f70 tx=0x7f2400009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.526+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f240c0f11e0 con 0x7f24100725e0 2024-03-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.526+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f240c0ecd80 con 0x7f24100725e0 2024-03-16T20:04:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.730+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2410068000 con 0x7f24100725e0 2024-03-16T20:04:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.742+0000 7f2405ffb640 1 -- 172.21.15.32:0/2009310430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f240c0b2070 con 0x7f24100725e0 2024-03-16T20:04:33.727 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:33.727 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 msgr2=0x7f23f4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 0x7f23f4069380 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24101f6f70 tx=0x7f2400009060 comp rx=0 tx=0).stop 2024-03-16T20:04:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 msgr2=0x7f2410142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f240c0b0040 tx=0x7f240c019130 comp rx=0 tx=0).stop 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 shutdown_connections 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f23f4066ec0 0x7f23f4069380 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f241015a420 0x7f24101530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2410150d70 0x7f2410152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 --2- 172.21.15.32:0/2009310430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24100725e0 0x7f2410142410 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 >> 172.21.15.32:0/2009310430 conn(0x7f241006c0d0 msgr2=0x7f24100c63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 shutdown_connections 2024-03-16T20:04:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:33.746+0000 7f241769f640 1 -- 172.21.15.32:0/2009310430 wait complete. 2024-03-16T20:04:34.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:34.350 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:35.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 Processor -- start 2024-03-16T20:04:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 -- start start 2024-03-16T20:04:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4f1c151000 0x7f4f1c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0c8ec0 con 0x7f4f1c0c36b0 2024-03-16T20:04:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c072950 con 0x7f4f1c0c8980 2024-03-16T20:04:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f20819640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c072f10 con 0x7f4f1c151000 2024-03-16T20:04:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f13fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4f1c0c36b0 0x7f4f1c0c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50324/0 (socket says 172.21.15.32:50324) 2024-03-16T20:04:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f13fff640 1 -- 172.21.15.32:0/1979985155 learned_addr learned my addr 172.21.15.32:0/1979985155 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.478+0000 7f4f1a59c640 1 --2- 172.21.15.32:0/1979985155 >> v2:172.21.15.59:3300/0 conn(0x7f4f1c151000 0x7f4f1c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:35.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.482+0000 7f4f19d9b640 1 -- 172.21.15.32:0/1979985155 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211495440 0 0) 0x7f4f1c0c8ec0 con 0x7f4f1c0c36b0 2024-03-16T20:04:35.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.482+0000 7f4f19d9b640 1 -- 172.21.15.32:0/1979985155 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f08003530 con 0x7f4f1c0c36b0 2024-03-16T20:04:35.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.486+0000 7f4f1a59c640 1 -- 172.21.15.32:0/1979985155 >> v1:172.21.15.59:6789/0 conn(0x7f4f1c0c8980 legacy=0x7f4f1c1615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:35.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.486+0000 7f4f1a59c640 1 -- 172.21.15.32:0/1979985155 >> v1:172.21.15.32:6789/0 conn(0x7f4f1c0c36b0 legacy=0x7f4f1c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:35.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.486+0000 7f4f1a59c640 1 -- 172.21.15.32:0/1979985155 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c072c30 con 0x7f4f1c151000 2024-03-16T20:04:35.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.490+0000 7f4f1a59c640 1 --2- 172.21.15.32:0/1979985155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c151000 0x7f4f1c0c8420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0097c0 tx=0x7f4f0c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5707c149a027db9d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:35.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.502+0000 7f4f19d9b640 1 -- 172.21.15.32:0/1979985155 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c004230 con 0x7f4f1c151000 2024-03-16T20:04:35.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.502+0000 7f4f19d9b640 1 -- 172.21.15.32:0/1979985155 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c0a7050 con 0x7f4f1c151000 2024-03-16T20:04:35.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.502+0000 7f4f19d9b640 1 -- 172.21.15.32:0/1979985155 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c003620 con 0x7f4f1c151000 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- 172.21.15.32:0/1979985155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c151000 msgr2=0x7f4f1c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 --2- 172.21.15.32:0/1979985155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c151000 0x7f4f1c0c8420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0097c0 tx=0x7f4f0c01c460 comp rx=0 tx=0).stop 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- 172.21.15.32:0/1979985155 shutdown_connections 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 --2- 172.21.15.32:0/1979985155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c151000 0x7f4f1c0c8420 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- 172.21.15.32:0/1979985155 >> 172.21.15.32:0/1979985155 conn(0x7f4f1c06c0d0 msgr2=0x7f4f1c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- 172.21.15.32:0/1979985155 shutdown_connections 2024-03-16T20:04:35.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- 172.21.15.32:0/1979985155 wait complete. 2024-03-16T20:04:35.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 Processor -- start 2024-03-16T20:04:35.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- start start 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c0c36b0 0x7f4f1c1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4f1c151000 0x7f4f1c14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c14cb40 con 0x7f4f1c0c8980 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c150020 con 0x7f4f1c0c36b0 2024-03-16T20:04:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f20819640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c150300 con 0x7f4f1c151000 2024-03-16T20:04:35.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f13fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:35.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f13fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41812/0 (socket says 172.21.15.32:41812) 2024-03-16T20:04:35.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.506+0000 7f4f13fff640 1 -- 172.21.15.32:0/3068978724 learned_addr learned my addr 172.21.15.32:0/3068978724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:35.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f1a59c640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c0c36b0 0x7f4f1c1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:35.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f1ad9d640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4f1c151000 0x7f4f1c14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f13fff640 1 -- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4f1c151000 msgr2=0x7f4f1c14f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f13fff640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4f1c151000 0x7f4f1c14f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f13fff640 1 -- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c0c36b0 msgr2=0x7f4f1c1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f13fff640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c0c36b0 0x7f4f1c1f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.510+0000 7f4f13fff640 1 -- 172.21.15.32:0/3068978724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c009420 con 0x7f4f1c0c8980 2024-03-16T20:04:35.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.514+0000 7f4f13fff640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4f04001210 tx=0x7f4f0400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.518+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04005cf0 con 0x7f4f1c0c8980 2024-03-16T20:04:35.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.518+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04005e50 con 0x7f4f1c0c8980 2024-03-16T20:04:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.522+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f040072d0 con 0x7f4f1c0c8980 2024-03-16T20:04:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.522+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c1505e0 con 0x7f4f1c0c8980 2024-03-16T20:04:35.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.522+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c147860 con 0x7f4f1c0c8980 2024-03-16T20:04:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.522+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c14c910 con 0x7f4f1c0c8980 2024-03-16T20:04:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.546+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4f04007430 con 0x7f4f1c0c8980 2024-03-16T20:04:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.546+0000 7f4f12ffd640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 0x7f4efc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.546+0000 7f4f1a59c640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 0x7f4efc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.546+0000 7f4f1a59c640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 0x7f4efc069300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0ad000 tx=0x7f4f0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.906+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4f04055130 con 0x7f4f1c0c8980 2024-03-16T20:04:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:35.910+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4f040513d0 con 0x7f4f1c0c8980 2024-03-16T20:04:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.110+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f1c14ccf0 con 0x7f4f1c0c8980 2024-03-16T20:04:36.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.130+0000 7f4f12ffd640 1 -- 172.21.15.32:0/3068978724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f4f1c14ccf0 con 0x7f4f1c0c8980 2024-03-16T20:04:36.115 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:36.115 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:36.116 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 msgr2=0x7f4efc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 0x7f4efc069300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0ad000 tx=0x7f4f0c008040 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 msgr2=0x7f4f1c14cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4f04001210 tx=0x7f4f0400c820 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 shutdown_connections 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4efc066e40 0x7f4efc069300 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4f1c151000 0x7f4f1c14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c0c8980 0x7f4f1c14cfd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 --2- 172.21.15.32:0/3068978724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f1c0c36b0 0x7f4f1c1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 >> 172.21.15.32:0/3068978724 conn(0x7f4f1c06c0d0 msgr2=0x7f4f1c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 shutdown_connections 2024-03-16T20:04:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:36.134+0000 7f4f20819640 1 -- 172.21.15.32:0/3068978724 wait complete. 2024-03-16T20:04:36.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:36.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:37.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:37.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.870+0000 7f889aaea640 1 Processor -- start 2024-03-16T20:04:37.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.870+0000 7f889aaea640 1 -- start start 2024-03-16T20:04:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f889aaea640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88940c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f889aaea640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940c8ec0 con 0x7f88940c8980 2024-03-16T20:04:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f889aaea640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894072950 con 0x7f88940c36b0 2024-03-16T20:04:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f889aaea640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8894072f10 con 0x7f8894151000 2024-03-16T20:04:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f8899087640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f88940c8980 0x7f88941615d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50328/0 (socket says 172.21.15.32:50328) 2024-03-16T20:04:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f8899087640 1 -- 172.21.15.32:0/2698166019 learned_addr learned my addr 172.21.15.32:0/2698166019 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.874+0000 7f8898886640 1 --2- 172.21.15.32:0/2698166019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88940c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:37.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.878+0000 7f88937fe640 1 -- 172.21.15.32:0/2698166019 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912327494 0 0) 0x7f88940c8ec0 con 0x7f88940c8980 2024-03-16T20:04:37.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.878+0000 7f88937fe640 1 -- 172.21.15.32:0/2698166019 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8884005740 con 0x7f88940c8980 2024-03-16T20:04:37.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.878+0000 7f8898886640 1 -- 172.21.15.32:0/2698166019 >> v1:172.21.15.59:6789/0 conn(0x7f88940c36b0 legacy=0x7f88940c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:37.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.878+0000 7f8898886640 1 -- 172.21.15.32:0/2698166019 >> v1:172.21.15.32:6789/0 conn(0x7f88940c8980 legacy=0x7f88941615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:37.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.878+0000 7f8898886640 1 -- 172.21.15.32:0/2698166019 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894072c30 con 0x7f8894151000 2024-03-16T20:04:37.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.882+0000 7f8898886640 1 --2- 172.21.15.32:0/2698166019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88940c8420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f88880097f0 tx=0x7f888801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce09095c2ea006a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:37.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f88937fe640 1 -- 172.21.15.32:0/2698166019 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88880ab600 con 0x7f8894151000 2024-03-16T20:04:37.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f88937fe640 1 -- 172.21.15.32:0/2698166019 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88880a7050 con 0x7f8894151000 2024-03-16T20:04:37.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f88937fe640 1 -- 172.21.15.32:0/2698166019 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88880ab980 con 0x7f8894151000 2024-03-16T20:04:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 -- 172.21.15.32:0/2698166019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 msgr2=0x7f88940c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 --2- 172.21.15.32:0/2698166019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88940c8420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f88880097f0 tx=0x7f888801c4e0 comp rx=0 tx=0).stop 2024-03-16T20:04:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 -- 172.21.15.32:0/2698166019 shutdown_connections 2024-03-16T20:04:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 --2- 172.21.15.32:0/2698166019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88940c8420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 -- 172.21.15.32:0/2698166019 >> 172.21.15.32:0/2698166019 conn(0x7f889406c0d0 msgr2=0x7f889406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 -- 172.21.15.32:0/2698166019 shutdown_connections 2024-03-16T20:04:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.894+0000 7f889aaea640 1 -- 172.21.15.32:0/2698166019 wait complete. 2024-03-16T20:04:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 Processor -- start 2024-03-16T20:04:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 -- start start 2024-03-16T20:04:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88940c36b0 0x7f8894156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88941f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889414a520 con 0x7f88940c8980 2024-03-16T20:04:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889414a7c0 con 0x7f88940c36b0 2024-03-16T20:04:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f889aaea640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88941f3040 con 0x7f8894151000 2024-03-16T20:04:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f8893fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f8893fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41816/0 (socket says 172.21.15.32:41816) 2024-03-16T20:04:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f8893fff640 1 -- 172.21.15.32:0/166091903 learned_addr learned my addr 172.21.15.32:0/166091903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:37.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.898+0000 7f8898886640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88940c36b0 0x7f8894156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:37.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.902+0000 7f8893fff640 1 -- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 msgr2=0x7f88941f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:37.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.902+0000 7f8893fff640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88941f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:37.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.902+0000 7f8893fff640 1 -- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88940c36b0 msgr2=0x7f8894156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:37.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.902+0000 7f8893fff640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88940c36b0 0x7f8894156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:37.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.902+0000 7f8893fff640 1 -- 172.21.15.32:0/166091903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888009420 con 0x7f88940c8980 2024-03-16T20:04:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.910+0000 7f8893fff640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f888000c420 tx=0x7f888000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.914+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800052d0 con 0x7f88940c8980 2024-03-16T20:04:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.914+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880005430 con 0x7f88940c8980 2024-03-16T20:04:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.914+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880007260 con 0x7f88940c8980 2024-03-16T20:04:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.914+0000 7f889aaea640 1 -- 172.21.15.32:0/166091903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88941f3320 con 0x7f88940c8980 2024-03-16T20:04:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.914+0000 7f889aaea640 1 -- 172.21.15.32:0/166091903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8894155200 con 0x7f88940c8980 2024-03-16T20:04:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.918+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c0052a0 con 0x7f88940c8980 2024-03-16T20:04:37.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.950+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f88800073c0 con 0x7f88940c8980 2024-03-16T20:04:37.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.954+0000 7f88917fa640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 0x7f8878069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:37.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.954+0000 7f8898886640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 0x7f8878069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:37.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.954+0000 7f8898886640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 0x7f8878069300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88880ad000 tx=0x7f88880a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:37.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:37.958+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8880007670 con 0x7f88940c8980 2024-03-16T20:04:38.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.230+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8880051500 con 0x7f88940c8980 2024-03-16T20:04:38.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.450+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f886c005900 con 0x7f88940c8980 2024-03-16T20:04:38.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f88917fa640 1 -- 172.21.15.32:0/166091903 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f886c005900 con 0x7f88940c8980 2024-03-16T20:04:38.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:38.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:38.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 msgr2=0x7f8878069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 0x7f8878069300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88880ad000 tx=0x7f88880a6000 comp rx=0 tx=0).stop 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 msgr2=0x7f88941f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f888000c420 tx=0x7f888000c8e0 comp rx=0 tx=0).stop 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 shutdown_connections 2024-03-16T20:04:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8878066e40 0x7f8878069300 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8894151000 0x7f88941f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88940c8980 0x7f88941f2770 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 --2- 172.21.15.32:0/166091903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88940c36b0 0x7f8894156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 >> 172.21.15.32:0/166091903 conn(0x7f889406c0d0 msgr2=0x7f889414b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 shutdown_connections 2024-03-16T20:04:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:38.474+0000 7f887effd640 1 -- 172.21.15.32:0/166091903 wait complete. 2024-03-16T20:04:39.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:39.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:40.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:40.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.205+0000 7fc389069640 1 Processor -- start 2024-03-16T20:04:40.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.205+0000 7fc389069640 1 -- start start 2024-03-16T20:04:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc389069640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc38406de40 0x7fc38415a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc389069640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc38406e220 0x7fc384164850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc389069640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38415a390 con 0x7fc3840c73f0 2024-03-16T20:04:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc389069640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc384071b70 con 0x7fc38406de40 2024-03-16T20:04:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc389069640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc384072bd0 con 0x7fc38406e220 2024-03-16T20:04:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc382d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc38406de40 0x7fc38415a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc38359e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc38406e220 0x7fc384164850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc38259c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc3840c73f0 0x7fc384153e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58424/0 (socket says 172.21.15.32:58424) 2024-03-16T20:04:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.209+0000 7fc38259c640 1 -- 172.21.15.32:0/3367393626 learned_addr learned my addr 172.21.15.32:0/3367393626 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.213+0000 7fc382d9d640 1 -- 172.21.15.32:0/3367393626 >> v2:172.21.15.59:3300/0 conn(0x7fc38406e220 msgr2=0x7fc384164850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.213+0000 7fc382d9d640 1 --2- 172.21.15.32:0/3367393626 >> v2:172.21.15.59:3300/0 conn(0x7fc38406e220 0x7fc384164850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.213+0000 7fc382d9d640 1 -- 172.21.15.32:0/3367393626 >> v1:172.21.15.32:6789/0 conn(0x7fc3840c73f0 legacy=0x7fc384153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.213+0000 7fc382d9d640 1 -- 172.21.15.32:0/3367393626 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384072eb0 con 0x7fc38406de40 2024-03-16T20:04:40.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.221+0000 7fc382d9d640 1 --2- 172.21.15.32:0/3367393626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406de40 0x7fc38415a820 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc3840c9e90 tx=0x7fc37401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd50125352ba64a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:40.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc381d9b640 1 -- 172.21.15.32:0/3367393626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3740a8970 con 0x7fc38406de40 2024-03-16T20:04:40.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc381d9b640 1 -- 172.21.15.32:0/3367393626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3740a8ad0 con 0x7fc38406de40 2024-03-16T20:04:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc381d9b640 1 -- 172.21.15.32:0/3367393626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3740a8e60 con 0x7fc38406de40 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 -- 172.21.15.32:0/3367393626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406de40 msgr2=0x7fc38415a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 --2- 172.21.15.32:0/3367393626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406de40 0x7fc38415a820 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc3840c9e90 tx=0x7fc37401ec70 comp rx=0 tx=0).stop 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 -- 172.21.15.32:0/3367393626 shutdown_connections 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 --2- 172.21.15.32:0/3367393626 >> v2:172.21.15.59:3300/0 conn(0x7fc38406e220 0x7fc384164850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 --2- 172.21.15.32:0/3367393626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406de40 0x7fc38415a820 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 -- 172.21.15.32:0/3367393626 >> 172.21.15.32:0/3367393626 conn(0x7fc3840cbfe0 msgr2=0x7fc3840cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 -- 172.21.15.32:0/3367393626 shutdown_connections 2024-03-16T20:04:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.225+0000 7fc389069640 1 -- 172.21.15.32:0/3367393626 wait complete. 2024-03-16T20:04:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 Processor -- start 2024-03-16T20:04:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 -- start start 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc38406de40 0x7fc3841502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3840afe20 0x7fc38414c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3840aecd0 con 0x7fc38406e220 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3841ef290 con 0x7fc3840afe20 2024-03-16T20:04:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc389069640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3841f2780 con 0x7fc38406de40 2024-03-16T20:04:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc38259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc38359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3840afe20 0x7fc38414c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc38259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56778/0 (socket says 172.21.15.32:56778) 2024-03-16T20:04:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc38259c640 1 -- 172.21.15.32:0/3031943702 learned_addr learned my addr 172.21.15.32:0/3031943702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.229+0000 7fc382d9d640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc38406de40 0x7fc3841502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.233+0000 7fc38259c640 1 -- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc38406de40 msgr2=0x7fc3841502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.233+0000 7fc38259c640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc38406de40 0x7fc3841502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.233+0000 7fc38259c640 1 -- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3840afe20 msgr2=0x7fc38414c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.233+0000 7fc38259c640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3840afe20 0x7fc38414c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.233+0000 7fc38259c640 1 -- 172.21.15.32:0/3031943702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374009420 con 0x7fc38406e220 2024-03-16T20:04:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.237+0000 7fc38259c640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c530 tx=0x7fc37800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:40.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378005cf0 con 0x7fc38406e220 2024-03-16T20:04:40.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc378005e50 con 0x7fc38406e220 2024-03-16T20:04:40.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3841578f0 con 0x7fc38406e220 2024-03-16T20:04:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378007270 con 0x7fc38406e220 2024-03-16T20:04:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3841f2d20 con 0x7fc38406e220 2024-03-16T20:04:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.245+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3840674e0 con 0x7fc38406e220 2024-03-16T20:04:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.533+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc3780073d0 con 0x7fc38406e220 2024-03-16T20:04:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.533+0000 7fc36b7fe640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 0x7fc370069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.533+0000 7fc382d9d640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 0x7fc370069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.533+0000 7fc382d9d640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 0x7fc370069330 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc3740b2000 tx=0x7fc374008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.545+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc378054f00 con 0x7fc38406e220 2024-03-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.545+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc3780511a0 con 0x7fc38406e220 2024-03-16T20:04:40.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.789+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc384068000 con 0x7fc38406e220 2024-03-16T20:04:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.881+0000 7fc36b7fe640 1 -- 172.21.15.32:0/3031943702 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc378017020 con 0x7fc38406e220 2024-03-16T20:04:40.866 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:40.867 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 msgr2=0x7fc370069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 0x7fc370069330 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc3740b2000 tx=0x7fc374008040 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 msgr2=0x7fc38414bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c530 tx=0x7fc37800c9f0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 shutdown_connections 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc370066e70 0x7fc370069330 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3840afe20 0x7fc38414c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc38406e220 0x7fc38414bad0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 --2- 172.21.15.32:0/3031943702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc38406de40 0x7fc3841502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 >> 172.21.15.32:0/3031943702 conn(0x7fc3840cbfe0 msgr2=0x7fc38406b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 shutdown_connections 2024-03-16T20:04:40.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:40.885+0000 7fc389069640 1 -- 172.21.15.32:0/3031943702 wait complete. 2024-03-16T20:04:41.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:41.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:42.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:04:42.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:04:42.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:42.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.613+0000 7f6809495640 1 Processor -- start 2024-03-16T20:04:42.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.613+0000 7f6809495640 1 -- start start 2024-03-16T20:04:42.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f6809495640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6804073010 0x7f68041658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:42.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f6809495640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68041546a0 con 0x7f68041542c0 2024-03-16T20:04:42.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f6809495640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804159f20 con 0x7f6804070360 2024-03-16T20:04:42.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f6809495640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680415a1e0 con 0x7f6804073010 2024-03-16T20:04:42.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f68037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6804073010 0x7f68041658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:42.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f68037fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6804073010 0x7f68041658b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56788/0 (socket says 172.21.15.32:56788) 2024-03-16T20:04:42.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.617+0000 7f68037fe640 1 -- 172.21.15.32:0/1904291580 learned_addr learned my addr 172.21.15.32:0/1904291580 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:42.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.621+0000 7f68037fe640 1 -- 172.21.15.32:0/1904291580 >> v1:172.21.15.59:6789/0 conn(0x7f6804070360 legacy=0x7f680415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:42.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.621+0000 7f68037fe640 1 -- 172.21.15.32:0/1904291580 >> v1:172.21.15.32:6789/0 conn(0x7f68041542c0 legacy=0x7f6804072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:42.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.621+0000 7f68037fe640 1 -- 172.21.15.32:0/1904291580 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680415a4c0 con 0x7f6804073010 2024-03-16T20:04:42.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.625+0000 7f68037fe640 1 --2- 172.21.15.32:0/1904291580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804073010 0x7f68041658b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009e50 tx=0x7f67fc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72512b9e9506769 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:42.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.629+0000 7f6801ffb640 1 -- 172.21.15.32:0/1904291580 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0a57f0 con 0x7f6804073010 2024-03-16T20:04:42.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.629+0000 7f6801ffb640 1 -- 172.21.15.32:0/1904291580 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc0a5950 con 0x7f6804073010 2024-03-16T20:04:42.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.629+0000 7f6801ffb640 1 -- 172.21.15.32:0/1904291580 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0a5ce0 con 0x7f6804073010 2024-03-16T20:04:42.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- 172.21.15.32:0/1904291580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804073010 msgr2=0x7f68041658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:42.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 --2- 172.21.15.32:0/1904291580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804073010 0x7f68041658b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009e50 tx=0x7f67fc01ca30 comp rx=0 tx=0).stop 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- 172.21.15.32:0/1904291580 shutdown_connections 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 --2- 172.21.15.32:0/1904291580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804073010 0x7f68041658b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- 172.21.15.32:0/1904291580 >> 172.21.15.32:0/1904291580 conn(0x7f680406c0d0 msgr2=0x7f680406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- 172.21.15.32:0/1904291580 shutdown_connections 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- 172.21.15.32:0/1904291580 wait complete. 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 Processor -- start 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- start start 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6804073010 0x7f680415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68041542c0 0x7f6804153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804143cc0 con 0x7f6804070360 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68041599d0 con 0x7f68041542c0 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6809495640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680415ccd0 con 0x7f6804073010 2024-03-16T20:04:42.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.633+0000 7f6802ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.637+0000 7f68037fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68041542c0 0x7f6804153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.637+0000 7f6802ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56804/0 (socket says 172.21.15.32:56804) 2024-03-16T20:04:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.637+0000 7f6802ffd640 1 -- 172.21.15.32:0/699128256 learned_addr learned my addr 172.21.15.32:0/699128256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.637+0000 7f68027fc640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6804073010 0x7f680415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.641+0000 7f6802ffd640 1 -- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6804073010 msgr2=0x7f680415c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.641+0000 7f6802ffd640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6804073010 0x7f680415c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.641+0000 7f6802ffd640 1 -- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68041542c0 msgr2=0x7f6804153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.641+0000 7f6802ffd640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68041542c0 0x7f6804153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.641+0000 7f6802ffd640 1 -- 172.21.15.32:0/699128256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc009b30 con 0x7f6804070360 2024-03-16T20:04:42.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.645+0000 7f6802ffd640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c450 tx=0x7f67f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:42.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f40052d0 con 0x7f6804070360 2024-03-16T20:04:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4005430 con 0x7f6804070360 2024-03-16T20:04:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680415cf90 con 0x7f6804070360 2024-03-16T20:04:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4007260 con 0x7f6804070360 2024-03-16T20:04:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6804143670 con 0x7f6804070360 2024-03-16T20:04:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.649+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6804068000 con 0x7f6804070360 2024-03-16T20:04:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.673+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f67f40073c0 con 0x7f6804070360 2024-03-16T20:04:42.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.673+0000 7f67eb7fe640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 0x7f67f0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:42.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.673+0000 7f68027fc640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 0x7f67f0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:42.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.673+0000 7f68027fc640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 0x7f67f0069330 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f67f8007fd0 tx=0x7f67f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:42.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.941+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f67f4055bf0 con 0x7f6804070360 2024-03-16T20:04:42.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:42.941+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f67f4051e90 con 0x7f6804070360 2024-03-16T20:04:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.197+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6804165a50 con 0x7f6804070360 2024-03-16T20:04:43.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.213+0000 7f67eb7fe640 1 -- 172.21.15.32:0/699128256 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f67f4055550 con 0x7f6804070360 2024-03-16T20:04:43.198 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:43.199 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 msgr2=0x7f67f0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 0x7f67f0069330 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f67f8007fd0 tx=0x7f67f8009040 comp rx=0 tx=0).stop 2024-03-16T20:04:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 msgr2=0x7f680415b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c450 tx=0x7f67f400c910 comp rx=0 tx=0).stop 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 shutdown_connections 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f67f0066e70 0x7f67f0069330 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68041542c0 0x7f6804153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6804073010 0x7f680415c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 --2- 172.21.15.32:0/699128256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6804070360 0x7f680415b0e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 >> 172.21.15.32:0/699128256 conn(0x7f680406c0d0 msgr2=0x7f6804142060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 shutdown_connections 2024-03-16T20:04:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:43.217+0000 7f6809495640 1 -- 172.21.15.32:0/699128256 wait complete. 2024-03-16T20:04:43.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:43.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:43.962 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:28.051 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.046 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:28.135 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.206 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:28.231 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.210 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:28.236 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.322 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:28.348 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.328 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:28.353 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.532 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:28.621 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:44.903 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:28.929 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.912 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:28.938 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:04:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 Processor -- start 2024-03-16T20:04:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 -- start start 2024-03-16T20:04:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80703d0 0x7fabf815d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:44.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf815a110 con 0x7fabf8154010 2024-03-16T20:04:44.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf815a3f0 con 0x7fabf8070a70 2024-03-16T20:04:44.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:44.997+0000 7fabfef98640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8072550 con 0x7fabf80703d0 2024-03-16T20:04:44.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.001+0000 7fabfcd34640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fabf8154010 0x7fabf8072c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58442/0 (socket says 172.21.15.32:58442) 2024-03-16T20:04:44.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.001+0000 7fabfcd34640 1 -- 172.21.15.32:0/878527733 learned_addr learned my addr 172.21.15.32:0/878527733 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.005+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661345274 0 0) 0x7fabf815a110 con 0x7fabf8154010 2024-03-16T20:04:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.005+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8003530 con 0x7fabf8154010 2024-03-16T20:04:45.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862553510 0 0) 0x7fabe8003530 con 0x7fabf8154010 2024-03-16T20:04:45.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80703d0 msgr2=0x7fabf815d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80703d0 0x7fabf815d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 >> v1:172.21.15.59:6789/0 conn(0x7fabf8070a70 legacy=0x7fabf8164720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf815a6d0 con 0x7fabf8154010 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fabec001680 con 0x7fabf8154010 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 >> v1:172.21.15.32:6789/0 conn(0x7fabf8154010 legacy=0x7fabf8072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80039b0 0x7fabe8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabe80047d0 0x7fabe8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabe80052b0 0x7fabe800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800c120 con 0x7fabe80047d0 2024-03-16T20:04:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800c290 con 0x7fabe80039b0 2024-03-16T20:04:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.017+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800c450 con 0x7fabe80052b0 2024-03-16T20:04:45.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.025+0000 7fabf7fff640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabe80047d0 0x7fabe8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.033+0000 7fabf7fff640 1 -- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabe80052b0 msgr2=0x7fabe800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.033+0000 7fabf7fff640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabe80052b0 0x7fabe800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.033+0000 7fabf7fff640 1 -- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80039b0 msgr2=0x7fabe8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.033+0000 7fabf7fff640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80039b0 0x7fabe8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.033+0000 7fabf7fff640 1 -- 172.21.15.32:0/878527733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe8003530 con 0x7fabe80047d0 2024-03-16T20:04:45.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.037+0000 7fabf7fff640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabe80047d0 0x7fabe8004c50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fabf80c3700 tx=0x7fabe401c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db2c649f1278c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40ab600 con 0x7fabe80047d0 2024-03-16T20:04:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe40a7050 con 0x7fabe80047d0 2024-03-16T20:04:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabf77fe640 1 -- 172.21.15.32:0/878527733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40ab9a0 con 0x7fabe80047d0 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- 172.21.15.32:0/878527733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabe80047d0 msgr2=0x7fabe8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 --2- 172.21.15.32:0/878527733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabe80047d0 0x7fabe8004c50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fabf80c3700 tx=0x7fabe401c220 comp rx=0 tx=0).stop 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfd535640 1 -- 172.21.15.32:0/878527733 reap_dead start 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- 172.21.15.32:0/878527733 shutdown_connections 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- 172.21.15.32:0/878527733 >> 172.21.15.32:0/878527733 conn(0x7fabf806c0d0 msgr2=0x7fabf806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- 172.21.15.32:0/878527733 shutdown_connections 2024-03-16T20:04:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- 172.21.15.32:0/878527733 wait complete. 2024-03-16T20:04:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 Processor -- start 2024-03-16T20:04:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.045+0000 7fabfef98640 1 -- start start 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80047d0 0x7fabf81517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabf814ec80 0x7fabf80c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8152460 con 0x7fabf814ec80 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf814e3c0 con 0x7fabe80047d0 2024-03-16T20:04:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.049+0000 7fabfef98640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81efa20 con 0x7fabf80cb620 2024-03-16T20:04:45.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.057+0000 7fabfd535640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:45.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.061+0000 7fabfd535640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:48054/0 (socket says 172.21.15.32:48054) 2024-03-16T20:04:45.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.061+0000 7fabfd535640 1 -- 172.21.15.32:0/2517414809 learned_addr learned my addr 172.21.15.32:0/2517414809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.065+0000 7fabfcd34640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80047d0 0x7fabf81517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.065+0000 7fabf7fff640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabf814ec80 0x7fabf80c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.073+0000 7fabfd535640 1 -- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80047d0 msgr2=0x7fabf81517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.073+0000 7fabfd535640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80047d0 0x7fabf81517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.073+0000 7fabfd535640 1 -- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabf814ec80 msgr2=0x7fabf80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.073+0000 7fabfd535640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabf814ec80 0x7fabf80c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.073+0000 7fabfd535640 1 -- 172.21.15.32:0/2517414809 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf81efc10 con 0x7fabf80cb620 2024-03-16T20:04:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.089+0000 7fabfd535640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fabf000c4f0 tx=0x7fabf000c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.101+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0007110 con 0x7fabf80cb620 2024-03-16T20:04:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.101+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf0007290 con 0x7fabf80cb620 2024-03-16T20:04:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.101+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0007640 con 0x7fabf80cb620 2024-03-16T20:04:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.101+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8151ce0 con 0x7fabf80cb620 2024-03-16T20:04:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.101+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81f01b0 con 0x7fabf80cb620 2024-03-16T20:04:45.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.105+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf81525a0 con 0x7fabf80cb620 2024-03-16T20:04:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.409+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fabf00077a0 con 0x7fabf80cb620 2024-03-16T20:04:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.409+0000 7fabf57fa640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 0x7fabdc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.409+0000 7fabfcd34640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 0x7fabdc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.409+0000 7fabfcd34640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 0x7fabdc069350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fabec004bb0 tx=0x7fabec001aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.437+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fabf0054c60 con 0x7fabf80cb620 2024-03-16T20:04:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.457+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fabf0050f00 con 0x7fabf80cb620 2024-03-16T20:04:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.681+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf81525a0 con 0x7fabf80cb620 2024-03-16T20:04:45.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.697+0000 7fabf57fa640 1 -- 172.21.15.32:0/2517414809 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fabf81525a0 con 0x7fabf80cb620 2024-03-16T20:04:45.682 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:45.682 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 msgr2=0x7fabdc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 0x7fabdc069350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fabec004bb0 tx=0x7fabec001aa0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 msgr2=0x7fabf80cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fabf000c4f0 tx=0x7fabf000c9b0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 shutdown_connections 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fabdc066e90 0x7fabdc069350 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fabf80cb620 0x7fabf80cbab0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabf814ec80 0x7fabf80c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 --2- 172.21.15.32:0/2517414809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe80047d0 0x7fabf81517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 >> 172.21.15.32:0/2517414809 conn(0x7fabf806c0d0 msgr2=0x7fabf80cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 shutdown_connections 2024-03-16T20:04:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:45.701+0000 7fabfef98640 1 -- 172.21.15.32:0/2517414809 wait complete. 2024-03-16T20:04:46.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:46.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:47.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:47.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.437+0000 7fc7b4f55640 1 Processor -- start 2024-03-16T20:04:47.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.437+0000 7fc7b4f55640 1 -- start start 2024-03-16T20:04:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7b4f55640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7b4f55640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7b015b2a0 0x7fc7b015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7b4f55640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b0158c80 con 0x7fc7b0151ef0 2024-03-16T20:04:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7b4f55640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b00b0ad0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7b4f55640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0157cb0 con 0x7fc7b00c81e0 2024-03-16T20:04:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7add9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7aed9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7b015b2a0 0x7fc7b015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7add9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:48056/0 (socket says 172.21.15.32:48056) 2024-03-16T20:04:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7add9b640 1 -- 172.21.15.32:0/740997146 learned_addr learned my addr 172.21.15.32:0/740997146 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.441+0000 7fc7ae59c640 1 --1- 172.21.15.32:0/740997146 >> v1:172.21.15.59:6789/0 conn(0x7fc7b0151ef0 0x7fc7b00c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:45650/0 (socket says 172.21.15.32:45650) 2024-03-16T20:04:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.445+0000 7fc7add9b640 1 -- 172.21.15.32:0/740997146 >> v2:172.21.15.59:3300/0 conn(0x7fc7b015b2a0 msgr2=0x7fc7b015b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.445+0000 7fc7add9b640 1 --2- 172.21.15.32:0/740997146 >> v2:172.21.15.59:3300/0 conn(0x7fc7b015b2a0 0x7fc7b015b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.445+0000 7fc7add9b640 1 -- 172.21.15.32:0/740997146 >> v1:172.21.15.59:6789/0 conn(0x7fc7b0151ef0 legacy=0x7fc7b00c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.445+0000 7fc7add9b640 1 -- 172.21.15.32:0/740997146 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b015bfc0 con 0x7fc7b00c81e0 2024-03-16T20:04:47.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.449+0000 7fc7add9b640 1 --2- 172.21.15.32:0/740997146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf2cf36aa3b85cb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:47.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7ad59a640 1 -- 172.21.15.32:0/740997146 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40a8a50 con 0x7fc7b00c81e0 2024-03-16T20:04:47.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7ad59a640 1 -- 172.21.15.32:0/740997146 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a40a8bb0 con 0x7fc7b00c81e0 2024-03-16T20:04:47.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7ad59a640 1 -- 172.21.15.32:0/740997146 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40b0440 con 0x7fc7b00c81e0 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- 172.21.15.32:0/740997146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 msgr2=0x7fc7b00c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- 172.21.15.32:0/740997146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a401ed50 comp rx=0 tx=0).stop 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- 172.21.15.32:0/740997146 shutdown_connections 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- 172.21.15.32:0/740997146 >> v2:172.21.15.59:3300/0 conn(0x7fc7b015b2a0 0x7fc7b015b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- 172.21.15.32:0/740997146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b00c8600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- 172.21.15.32:0/740997146 >> 172.21.15.32:0/740997146 conn(0x7fc7b0070f90 msgr2=0x7fc7b00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- 172.21.15.32:0/740997146 shutdown_connections 2024-03-16T20:04:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- 172.21.15.32:0/740997146 wait complete. 2024-03-16T20:04:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 Processor -- start 2024-03-16T20:04:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- start start 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b0149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b0151ef0 0x7fc7b01595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00b18e0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00c33e0 con 0x7fc7b0151ef0 2024-03-16T20:04:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.461+0000 7fc7b4f55640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0158e70 con 0x7fc7b00c81e0 2024-03-16T20:04:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.465+0000 7fc7aed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.465+0000 7fc7add9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b0151ef0 0x7fc7b01595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.465+0000 7fc7aed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56842/0 (socket says 172.21.15.32:56842) 2024-03-16T20:04:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.465+0000 7fc7aed9d640 1 -- 172.21.15.32:0/3316320797 learned_addr learned my addr 172.21.15.32:0/3316320797 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:47.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 -- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 msgr2=0x7fc7b0149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:47.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b0149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:47.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 -- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b0151ef0 msgr2=0x7fc7b01595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:47.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b0151ef0 0x7fc7b01595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:47.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 -- 172.21.15.32:0/3316320797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40094d0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.469+0000 7fc7aed9d640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009d30 tx=0x7fc7a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:47.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.473+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8009ba0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.473+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a800f040 con 0x7fc7b015b2a0 2024-03-16T20:04:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.473+0000 7fc7b4f55640 1 -- 172.21.15.32:0/3316320797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b01ee5c0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.473+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80146c0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.473+0000 7fc7b4f55640 1 -- 172.21.15.32:0/3316320797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b00c3910 con 0x7fc7b015b2a0 2024-03-16T20:04:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.477+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7880052a0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.513+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc7a8014820 con 0x7fc7b015b2a0 2024-03-16T20:04:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.513+0000 7fc79effd640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 0x7fc790069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.513+0000 7fc7ae59c640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 0x7fc790069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.513+0000 7fc7ae59c640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 0x7fc790069350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0000f70 tx=0x7fc7a0007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.517+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc7a8056f00 con 0x7fc7b015b2a0 2024-03-16T20:04:47.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:47.809+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7a8052aa0 con 0x7fc7b015b2a0 2024-03-16T20:04:47.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.005+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc788005900 con 0x7fc7b015b2a0 2024-03-16T20:04:48.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.025+0000 7fc79effd640 1 -- 172.21.15.32:0/3316320797 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc788005900 con 0x7fc7b015b2a0 2024-03-16T20:04:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:48.012 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:48.012 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 msgr2=0x7fc790069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 0x7fc790069350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0000f70 tx=0x7fc7a0007a10 comp rx=0 tx=0).stop 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 msgr2=0x7fc7b0159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009d30 tx=0x7fc7a800c490 comp rx=0 tx=0).stop 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 shutdown_connections 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc790066e90 0x7fc790069350 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7b015b2a0 0x7fc7b0159b20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b0151ef0 0x7fc7b01595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 --2- 172.21.15.32:0/3316320797 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7b00c81e0 0x7fc7b0149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 >> 172.21.15.32:0/3316320797 conn(0x7fc7b0070f90 msgr2=0x7fc7b00ca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 shutdown_connections 2024-03-16T20:04:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:48.029+0000 7fc79cff9640 1 -- 172.21.15.32:0/3316320797 wait complete. 2024-03-16T20:04:48.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:48.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:49.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.761+0000 7f42cb87a640 1 Processor -- start 2024-03-16T20:04:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.761+0000 7f42cb87a640 1 -- start start 2024-03-16T20:04:49.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42cb87a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42c4070840 0x7f42c40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42cb87a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c4070c90 con 0x7f42c40724a0 2024-03-16T20:04:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42cb87a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c40c8590 con 0x7f42c4153fa0 2024-03-16T20:04:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42cb87a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c40c3f00 con 0x7f42c4070840 2024-03-16T20:04:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42c9e17640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f42c40724a0 0x7f42c4164770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47934/0 (socket says 172.21.15.32:47934) 2024-03-16T20:04:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42c9e17640 1 -- 172.21.15.32:0/434961706 learned_addr learned my addr 172.21.15.32:0/434961706 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.765+0000 7f42c8e15640 1 --2- 172.21.15.32:0/434961706 >> v2:172.21.15.32:3300/0 conn(0x7f42c4070840 0x7f42c40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.773+0000 7f42c8e15640 1 -- 172.21.15.32:0/434961706 >> v1:172.21.15.59:6789/0 conn(0x7f42c4153fa0 legacy=0x7f42c415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.773+0000 7f42c8e15640 1 -- 172.21.15.32:0/434961706 >> v1:172.21.15.32:6789/0 conn(0x7f42c40724a0 legacy=0x7f42c4164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.773+0000 7f42c8e15640 1 -- 172.21.15.32:0/434961706 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c4073590 con 0x7f42c4070840 2024-03-16T20:04:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.777+0000 7f42c8e15640 1 --2- 172.21.15.32:0/434961706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c4070840 0x7f42c40c7470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f42b40097c0 tx=0x7f42b401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d858ffbe15c27e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42bbfff640 1 -- 172.21.15.32:0/434961706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a8990 con 0x7f42c4070840 2024-03-16T20:04:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42bbfff640 1 -- 172.21.15.32:0/434961706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40a8af0 con 0x7f42c4070840 2024-03-16T20:04:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42bbfff640 1 -- 172.21.15.32:0/434961706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a8e80 con 0x7f42c4070840 2024-03-16T20:04:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 -- 172.21.15.32:0/434961706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c4070840 msgr2=0x7f42c40c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 --2- 172.21.15.32:0/434961706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c4070840 0x7f42c40c7470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f42b40097c0 tx=0x7f42b401ec90 comp rx=0 tx=0).stop 2024-03-16T20:04:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 -- 172.21.15.32:0/434961706 shutdown_connections 2024-03-16T20:04:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 --2- 172.21.15.32:0/434961706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c4070840 0x7f42c40c7470 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 -- 172.21.15.32:0/434961706 >> 172.21.15.32:0/434961706 conn(0x7f42c406c0d0 msgr2=0x7f42c406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 -- 172.21.15.32:0/434961706 shutdown_connections 2024-03-16T20:04:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.781+0000 7f42cb87a640 1 -- 172.21.15.32:0/434961706 wait complete. 2024-03-16T20:04:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 Processor -- start 2024-03-16T20:04:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 -- start start 2024-03-16T20:04:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c4070840 0x7f42c4146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f42c4153fa0 0x7f42c41421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4142c70 con 0x7f42c40724a0 2024-03-16T20:04:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41e5870 con 0x7f42c4070840 2024-03-16T20:04:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42cb87a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41e4610 con 0x7f42c4153fa0 2024-03-16T20:04:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42c8e15640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42c8e15640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49308/0 (socket says 172.21.15.32:49308) 2024-03-16T20:04:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42c8e15640 1 -- 172.21.15.32:0/2682262902 learned_addr learned my addr 172.21.15.32:0/2682262902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42c9616640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c4070840 0x7f42c4146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:49.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.785+0000 7f42c9e17640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f42c4153fa0 0x7f42c41421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:49.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 -- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f42c4153fa0 msgr2=0x7f42c41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:49.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f42c4153fa0 0x7f42c41421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:49.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 -- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c4070840 msgr2=0x7f42c4146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:49.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c4070840 0x7f42c4146f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:49.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 -- 172.21.15.32:0/2682262902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4009420 con 0x7f42c40724a0 2024-03-16T20:04:49.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.793+0000 7f42c8e15640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b3040 tx=0x7f42b401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:49.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b9630 con 0x7f42c40724a0 2024-03-16T20:04:49.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40b97b0 con 0x7f42c40724a0 2024-03-16T20:04:49.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c4142e80 con 0x7f42c40724a0 2024-03-16T20:04:49.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b9b60 con 0x7f42c40724a0 2024-03-16T20:04:49.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c41e35f0 con 0x7f42c40724a0 2024-03-16T20:04:49.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.801+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c41e3450 con 0x7f42c40724a0 2024-03-16T20:04:49.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.833+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f42b40b7040 con 0x7f42c40724a0 2024-03-16T20:04:49.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.833+0000 7f42b9ffb640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 0x7f42a8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:49.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.833+0000 7f42c9616640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 0x7f42a8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:49.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.833+0000 7f42c9616640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 0x7f42a8069350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f42c415a7a0 tx=0x7f42c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:49.837+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f42b40f4040 con 0x7f42c40724a0 2024-03-16T20:04:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.117+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f42b40f02e0 con 0x7f42c40724a0 2024-03-16T20:04:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.321+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42c41e3450 con 0x7f42c40724a0 2024-03-16T20:04:50.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.345+0000 7f42b9ffb640 1 -- 172.21.15.32:0/2682262902 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f42c41e3450 con 0x7f42c40724a0 2024-03-16T20:04:50.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:50.330 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 msgr2=0x7f42a8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 0x7f42a8069350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f42c415a7a0 tx=0x7f42c0009040 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 msgr2=0x7f42c41e40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b3040 tx=0x7f42b401ba30 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 shutdown_connections 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f42a8066e90 0x7f42a8069350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f42c4153fa0 0x7f42c41421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c40724a0 0x7f42c41e40d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 --2- 172.21.15.32:0/2682262902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c4070840 0x7f42c4146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 >> 172.21.15.32:0/2682262902 conn(0x7f42c406c0d0 msgr2=0x7f42c414e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 shutdown_connections 2024-03-16T20:04:50.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:50.349+0000 7f42cb87a640 1 -- 172.21.15.32:0/2682262902 wait complete. 2024-03-16T20:04:50.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:50.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:51.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:52.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 Processor -- start 2024-03-16T20:04:52.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 -- start start 2024-03-16T20:04:52.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594072c70 0x7f75940c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7594073050 0x7f75940c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759406e6a0 con 0x7f759406bae0 2024-03-16T20:04:52.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7594156f20 con 0x7f7594073050 2024-03-16T20:04:52.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759be82640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759406c250 con 0x7f7594072c70 2024-03-16T20:04:52.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759941d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7594073050 0x7f75940c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759941d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7594073050 0x7f75940c14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49310/0 (socket says 172.21.15.32:49310) 2024-03-16T20:04:52.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f759941d640 1 -- 172.21.15.32:0/1355066760 learned_addr learned my addr 172.21.15.32:0/1355066760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:52.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.089+0000 7f7599c1e640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594072c70 0x7f75940c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.093+0000 7f759941d640 1 -- 172.21.15.32:0/1355066760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594072c70 msgr2=0x7f75940c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:52.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.093+0000 7f759941d640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594072c70 0x7f75940c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.093+0000 7f759941d640 1 -- 172.21.15.32:0/1355066760 >> v1:172.21.15.59:6789/0 conn(0x7f759406bae0 legacy=0x7f7594150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.093+0000 7f759941d640 1 -- 172.21.15.32:0/1355066760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759406c510 con 0x7f7594073050 2024-03-16T20:04:52.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.097+0000 7f759941d640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7594073050 0x7f75940c14d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7584009870 tx=0x7f758401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff062059da88bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f7598c1c640 1 -- 172.21.15.32:0/1355066760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840a9610 con 0x7f7594073050 2024-03-16T20:04:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f7598c1c640 1 -- 172.21.15.32:0/1355066760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75840a9770 con 0x7f7594073050 2024-03-16T20:04:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f7598c1c640 1 -- 172.21.15.32:0/1355066760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840a9b00 con 0x7f7594073050 2024-03-16T20:04:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 -- 172.21.15.32:0/1355066760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7594073050 msgr2=0x7f75940c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7594073050 0x7f75940c14d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7584009870 tx=0x7f758401c530 comp rx=0 tx=0).stop 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 -- 172.21.15.32:0/1355066760 shutdown_connections 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7594073050 0x7f75940c14d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 --2- 172.21.15.32:0/1355066760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594072c70 0x7f75940c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 -- 172.21.15.32:0/1355066760 >> 172.21.15.32:0/1355066760 conn(0x7f759406afc0 msgr2=0x7f759406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 -- 172.21.15.32:0/1355066760 shutdown_connections 2024-03-16T20:04:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.101+0000 7f759be82640 1 -- 172.21.15.32:0/1355066760 wait complete. 2024-03-16T20:04:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 Processor -- start 2024-03-16T20:04:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 -- start start 2024-03-16T20:04:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 0x7f75940c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 0x7f759414bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594073050 0x7f75941495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759414a950 con 0x7f759406bae0 2024-03-16T20:04:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75941f6fc0 con 0x7f7594072c70 2024-03-16T20:04:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759be82640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759414b220 con 0x7f7594073050 2024-03-16T20:04:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f7599c1e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 0x7f75940c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759941d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 0x7f759414bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f7599c1e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 0x7f75940c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49314/0 (socket says 172.21.15.32:49314) 2024-03-16T20:04:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f7599c1e640 1 -- 172.21.15.32:0/969588724 learned_addr learned my addr 172.21.15.32:0/969588724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.105+0000 7f759a41f640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594073050 0x7f75941495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.109+0000 7f759941d640 1 -- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594073050 msgr2=0x7f75941495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.109+0000 7f759941d640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594073050 0x7f75941495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.109+0000 7f759941d640 1 -- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 msgr2=0x7f75940c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.109+0000 7f759941d640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 0x7f75940c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.109+0000 7f759941d640 1 -- 172.21.15.32:0/969588724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840094d0 con 0x7f7594072c70 2024-03-16T20:04:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.113+0000 7f759941d640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 0x7f759414bbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f75840aa000 tx=0x7f75840192d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840b9620 con 0x7f7594072c70 2024-03-16T20:04:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75840b97c0 con 0x7f7594072c70 2024-03-16T20:04:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75941f60f0 con 0x7f7594072c70 2024-03-16T20:04:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840b9b70 con 0x7f7594072c70 2024-03-16T20:04:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75941f6620 con 0x7f7594072c70 2024-03-16T20:04:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.125+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75940674e0 con 0x7f7594072c70 2024-03-16T20:04:52.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.461+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f75840c3480 con 0x7f7594072c70 2024-03-16T20:04:52.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.461+0000 7f758a7fc640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 0x7f7578069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:52.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.461+0000 7f7599c1e640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 0x7f7578069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:52.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.465+0000 7f7599c1e640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 0x7f7578069300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7590009c20 tx=0x7f7590009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.473+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f75840efd60 con 0x7f7594072c70 2024-03-16T20:04:52.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.505+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f75840ec000 con 0x7f7594072c70 2024-03-16T20:04:52.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.705+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f759414dda0 con 0x7f7594072c70 2024-03-16T20:04:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.749+0000 7f758a7fc640 1 -- 172.21.15.32:0/969588724 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f75840a95d0 con 0x7f7594072c70 2024-03-16T20:04:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:52.734 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 msgr2=0x7f7578069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 0x7f7578069300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7590009c20 tx=0x7f7590009190 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 msgr2=0x7f759414bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 0x7f759414bbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f75840aa000 tx=0x7f75840192d0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 shutdown_connections 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7578066e40 0x7f7578069300 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7594073050 0x7f75941495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7594072c70 0x7f759414bbf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 --2- 172.21.15.32:0/969588724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f759406bae0 0x7f75940c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 >> 172.21.15.32:0/969588724 conn(0x7f759406afc0 msgr2=0x7f759414d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 shutdown_connections 2024-03-16T20:04:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:52.753+0000 7f759be82640 1 -- 172.21.15.32:0/969588724 wait complete. 2024-03-16T20:04:53.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:53.362 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:54.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:54.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.485+0000 7ff090944640 1 Processor -- start 2024-03-16T20:04:54.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.485+0000 7ff090944640 1 -- start start 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff090944640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff08c0c36b0 0x7ff08c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff090944640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0c8ec0 con 0x7ff08c0c8980 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff090944640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c072950 con 0x7ff08c151000 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff090944640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c072f10 con 0x7ff08c0c36b0 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff08ad9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff08c0c8980 0x7ff08c1615d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47946/0 (socket says 172.21.15.32:47946) 2024-03-16T20:04:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff08ad9d640 1 -- 172.21.15.32:0/479800680 learned_addr learned my addr 172.21.15.32:0/479800680 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:54.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.489+0000 7ff089d9b640 1 --2- 172.21.15.32:0/479800680 >> v2:172.21.15.59:3300/0 conn(0x7ff08c0c36b0 0x7ff08c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:54.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.493+0000 7ff08959a640 1 -- 172.21.15.32:0/479800680 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594611403 0 0) 0x7ff08c0c8ec0 con 0x7ff08c0c8980 2024-03-16T20:04:54.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.493+0000 7ff08959a640 1 -- 172.21.15.32:0/479800680 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff074003530 con 0x7ff08c0c8980 2024-03-16T20:04:54.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.497+0000 7ff089d9b640 1 -- 172.21.15.32:0/479800680 >> v1:172.21.15.59:6789/0 conn(0x7ff08c151000 legacy=0x7ff08c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:54.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.497+0000 7ff089d9b640 1 -- 172.21.15.32:0/479800680 >> v1:172.21.15.32:6789/0 conn(0x7ff08c0c8980 legacy=0x7ff08c1615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:54.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.497+0000 7ff089d9b640 1 -- 172.21.15.32:0/479800680 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c072c30 con 0x7ff08c0c36b0 2024-03-16T20:04:54.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.505+0000 7ff089d9b640 1 --2- 172.21.15.32:0/479800680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c0c8420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0800097c0 tx=0x7ff08001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f85596cb950bd9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:54.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff08959a640 1 -- 172.21.15.32:0/479800680 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800a8050 con 0x7ff08c0c36b0 2024-03-16T20:04:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff08959a640 1 -- 172.21.15.32:0/479800680 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0800ac5f0 con 0x7ff08c0c36b0 2024-03-16T20:04:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff08959a640 1 -- 172.21.15.32:0/479800680 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800ac980 con 0x7ff08c0c36b0 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff090944640 1 -- 172.21.15.32:0/479800680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 msgr2=0x7ff08c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff090944640 1 --2- 172.21.15.32:0/479800680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c0c8420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0800097c0 tx=0x7ff08001c4a0 comp rx=0 tx=0).stop 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff090944640 1 -- 172.21.15.32:0/479800680 shutdown_connections 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff090944640 1 --2- 172.21.15.32:0/479800680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c0c8420 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.517+0000 7ff090944640 1 -- 172.21.15.32:0/479800680 >> 172.21.15.32:0/479800680 conn(0x7ff08c06c0d0 msgr2=0x7ff08c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- 172.21.15.32:0/479800680 shutdown_connections 2024-03-16T20:04:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- 172.21.15.32:0/479800680 wait complete. 2024-03-16T20:04:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 Processor -- start 2024-03-16T20:04:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- start start 2024-03-16T20:04:54.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c14fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:54.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:54.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff08c151000 0x7ff08c1432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c14ce50 con 0x7ff08c0c8980 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c14bad0 con 0x7ff08c0c36b0 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff090944640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c14bdb0 con 0x7ff08c151000 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.521+0000 7ff089d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.525+0000 7ff089d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49318/0 (socket says 172.21.15.32:49318) 2024-03-16T20:04:54.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.525+0000 7ff089d9b640 1 -- 172.21.15.32:0/2338904953 learned_addr learned my addr 172.21.15.32:0/2338904953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:54.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.525+0000 7ff08ad9d640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff08c151000 0x7ff08c1432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 -- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff08c151000 msgr2=0x7ff08c1432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff08c151000 0x7ff08c1432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 -- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 msgr2=0x7ff08c14fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c14fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 -- 172.21.15.32:0/2338904953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080009420 con 0x7ff08c0c8980 2024-03-16T20:04:54.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.529+0000 7ff089d9b640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff08001a0d0 tx=0x7ff080019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:54.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.533+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800a8050 con 0x7ff08c0c8980 2024-03-16T20:04:54.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.533+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0800b5640 con 0x7ff08c0c8980 2024-03-16T20:04:54.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.533+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c14c090 con 0x7ff08c0c8980 2024-03-16T20:04:54.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.533+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800b5a10 con 0x7ff08c0c8980 2024-03-16T20:04:54.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.533+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c14c650 con 0x7ff08c0c8980 2024-03-16T20:04:54.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.537+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c068000 con 0x7ff08c0c8980 2024-03-16T20:04:54.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.553+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0800b3040 con 0x7ff08c0c8980 2024-03-16T20:04:54.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.557+0000 7ff07affd640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 0x7ff06c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:54.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.557+0000 7ff08a59c640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 0x7ff06c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:54.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.557+0000 7ff08a59c640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 0x7ff06c069300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff07c000f70 tx=0x7ff07c0079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:54.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.557+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff0800f0280 con 0x7ff08c0c8980 2024-03-16T20:04:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:54.813+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff0800ebe20 con 0x7ff08c0c8980 2024-03-16T20:04:55.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.017+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08c0c5110 con 0x7ff08c0c8980 2024-03-16T20:04:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.041+0000 7ff07affd640 1 -- 172.21.15.32:0/2338904953 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff0800b9090 con 0x7ff08c0c8980 2024-03-16T20:04:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:55.028 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 msgr2=0x7ff06c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 0x7ff06c069300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff07c000f70 tx=0x7ff07c0079e0 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 msgr2=0x7ff08c148c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff08001a0d0 tx=0x7ff080019240 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 shutdown_connections 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff06c066e40 0x7ff06c069300 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff08c151000 0x7ff08c1432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08c0c8980 0x7ff08c148c00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 --2- 172.21.15.32:0/2338904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff08c0c36b0 0x7ff08c14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 >> 172.21.15.32:0/2338904953 conn(0x7ff08c06c0d0 msgr2=0x7ff08c14d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 shutdown_connections 2024-03-16T20:04:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:55.045+0000 7ff090944640 1 -- 172.21.15.32:0/2338904953 wait complete. 2024-03-16T20:04:55.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:55.651 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:56.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.773+0000 7f88502d6640 1 Processor -- start 2024-03-16T20:04:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.773+0000 7f88502d6640 1 -- start start 2024-03-16T20:04:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.773+0000 7f88502d6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8848151000 0x7f88480c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.773+0000 7f88502d6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8848156b30 0x7f88480c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f88502d6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88480c7a40 con 0x7f88480c81b0 2024-03-16T20:04:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f88502d6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88480c52e0 con 0x7f8848151000 2024-03-16T20:04:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f88502d6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88480c9270 con 0x7f8848156b30 2024-03-16T20:04:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f884e072640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8848151000 0x7f88480c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f884e873640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f88480c81b0 0x7f88480c3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47950/0 (socket says 172.21.15.32:47950) 2024-03-16T20:04:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f884e873640 1 -- 172.21.15.32:0/194235261 learned_addr learned my addr 172.21.15.32:0/194235261 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:04:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.777+0000 7f884d871640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8848156b30 0x7f88480c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.781+0000 7f884e072640 1 -- 172.21.15.32:0/194235261 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8848156b30 msgr2=0x7f88480c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.781+0000 7f884e072640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8848156b30 0x7f88480c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.781+0000 7f884e072640 1 -- 172.21.15.32:0/194235261 >> v1:172.21.15.32:6789/0 conn(0x7f88480c81b0 legacy=0x7f88480c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.781+0000 7f884e072640 1 -- 172.21.15.32:0/194235261 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88480c9510 con 0x7f8848151000 2024-03-16T20:04:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.785+0000 7f884e072640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848151000 0x7f88480c70d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f88440097c0 tx=0x7f884401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d44bcfe629a3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f884d070640 1 -- 172.21.15.32:0/194235261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88440a8990 con 0x7f8848151000 2024-03-16T20:04:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f884d070640 1 -- 172.21.15.32:0/194235261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88440a8af0 con 0x7f8848151000 2024-03-16T20:04:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f884d070640 1 -- 172.21.15.32:0/194235261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88440a8e80 con 0x7f8848151000 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 -- 172.21.15.32:0/194235261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848151000 msgr2=0x7f88480c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848151000 0x7f88480c70d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f88440097c0 tx=0x7f884401ec90 comp rx=0 tx=0).stop 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 -- 172.21.15.32:0/194235261 shutdown_connections 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8848156b30 0x7f88480c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 --2- 172.21.15.32:0/194235261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848151000 0x7f88480c70d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 -- 172.21.15.32:0/194235261 >> 172.21.15.32:0/194235261 conn(0x7f884806c0d0 msgr2=0x7f884806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 -- 172.21.15.32:0/194235261 shutdown_connections 2024-03-16T20:04:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 -- 172.21.15.32:0/194235261 wait complete. 2024-03-16T20:04:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.789+0000 7f88502d6640 1 Processor -- start 2024-03-16T20:04:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 -- start start 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88480c81b0 0x7f88480cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 0x7f884815a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 0x7f884815a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884814db70 con 0x7f8848156b30 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88481edf90 con 0x7f8848151000 2024-03-16T20:04:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f88502d6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88480cce10 con 0x7f88480c81b0 2024-03-16T20:04:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f884d871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 0x7f884815a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f884e072640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88480c81b0 0x7f88480cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f884e873640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 0x7f884815a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f884d871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 0x7f884815a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:53670/0 (socket says 172.21.15.32:53670) 2024-03-16T20:04:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.793+0000 7f884d871640 1 -- 172.21.15.32:0/2044963488 learned_addr learned my addr 172.21.15.32:0/2044963488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 -- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88480c81b0 msgr2=0x7f88480cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88480c81b0 0x7f88480cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 -- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 msgr2=0x7f884815a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 0x7f884815a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 -- 172.21.15.32:0/2044963488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8844009420 con 0x7f8848156b30 2024-03-16T20:04:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.801+0000 7f884e873640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 0x7f884815a900 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f884000caa0 tx=0x7f884000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884000ab40 con 0x7f8848156b30 2024-03-16T20:04:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884000aca0 con 0x7f8848156b30 2024-03-16T20:04:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840012600 con 0x7f8848156b30 2024-03-16T20:04:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88481ee230 con 0x7f8848156b30 2024-03-16T20:04:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8848146b10 con 0x7f8848156b30 2024-03-16T20:04:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:56.809+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8848068000 con 0x7f8848156b30 2024-03-16T20:04:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.097+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8840007070 con 0x7f8848156b30 2024-03-16T20:04:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.097+0000 7f883effd640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 0x7f882c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.097+0000 7f884e072640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 0x7f882c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.097+0000 7f884e072640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 0x7f882c069330 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f88440b2000 tx=0x7f8844008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:57.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.113+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f88400557f0 con 0x7f8848156b30 2024-03-16T20:04:57.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.113+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8840051a90 con 0x7f8848156b30 2024-03-16T20:04:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.417+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88481478c0 con 0x7f8848156b30 2024-03-16T20:04:57.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.445+0000 7f883effd640 1 -- 172.21.15.32:0/2044963488 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f8840017090 con 0x7f8848156b30 2024-03-16T20:04:57.429 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:57.429 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:57.430 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 msgr2=0x7f882c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 0x7f882c069330 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f88440b2000 tx=0x7f8844008040 comp rx=0 tx=0).stop 2024-03-16T20:04:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 msgr2=0x7f884815a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 0x7f884815a900 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f884000caa0 tx=0x7f884000cf60 comp rx=0 tx=0).stop 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 shutdown_connections 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f882c066e70 0x7f882c069330 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8848156b30 0x7f884815a900 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8848151000 0x7f884815a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 --2- 172.21.15.32:0/2044963488 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88480c81b0 0x7f88480cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 >> 172.21.15.32:0/2044963488 conn(0x7f884806c0d0 msgr2=0x7f8848154740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 shutdown_connections 2024-03-16T20:04:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:57.449+0000 7f88502d6640 1 -- 172.21.15.32:0/2044963488 wait complete. 2024-03-16T20:04:58.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:04:58.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:04:59.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:04:59.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.177+0000 7f5862e1a640 1 Processor -- start 2024-03-16T20:04:59.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.177+0000 7f5862e1a640 1 -- start start 2024-03-16T20:04:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.177+0000 7f5862e1a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f585c072c70 0x7f585c0c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.177+0000 7f5862e1a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c06bae0 0x7f585c151330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5862e1a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585c06e6a0 con 0x7f585c073050 2024-03-16T20:04:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5862e1a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585c156f20 con 0x7f585c072c70 2024-03-16T20:04:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5862e1a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c06c250 con 0x7f585c06bae0 2024-03-16T20:04:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f585c072c70 0x7f585c0c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f585c072c70 0x7f585c0c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46314/0 (socket says 172.21.15.32:46314) 2024-03-16T20:04:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 -- 172.21.15.32:0/2999522532 learned_addr learned my addr 172.21.15.32:0/2999522532 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f58613b7640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c06bae0 0x7f585c151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 -- 172.21.15.32:0/2999522532 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c06bae0 msgr2=0x7f585c151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:04:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c06bae0 0x7f585c151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 -- 172.21.15.32:0/2999522532 >> v1:172.21.15.59:6789/0 conn(0x7f585c073050 legacy=0x7f585c150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.181+0000 7f5860bb6640 1 -- 172.21.15.32:0/2999522532 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c06c510 con 0x7f585c072c70 2024-03-16T20:04:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.185+0000 7f5860bb6640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c072c70 0x7f585c0c0f90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5850009870 tx=0x7f585001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6a3166f9fb601d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f585b7fe640 1 -- 172.21.15.32:0/2999522532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500a9610 con 0x7f585c072c70 2024-03-16T20:04:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f585b7fe640 1 -- 172.21.15.32:0/2999522532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58500a9770 con 0x7f585c072c70 2024-03-16T20:04:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f585b7fe640 1 -- 172.21.15.32:0/2999522532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500a9b00 con 0x7f585c072c70 2024-03-16T20:04:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- 172.21.15.32:0/2999522532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c072c70 msgr2=0x7f585c0c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c072c70 0x7f585c0c0f90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5850009870 tx=0x7f585001c530 comp rx=0 tx=0).stop 2024-03-16T20:04:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- 172.21.15.32:0/2999522532 shutdown_connections 2024-03-16T20:04:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c06bae0 0x7f585c151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- 172.21.15.32:0/2999522532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c072c70 0x7f585c0c0f90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- 172.21.15.32:0/2999522532 >> 172.21.15.32:0/2999522532 conn(0x7f585c06afc0 msgr2=0x7f585c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- 172.21.15.32:0/2999522532 shutdown_connections 2024-03-16T20:04:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- 172.21.15.32:0/2999522532 wait complete. 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 Processor -- start 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- start start 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f585c072c70 0x7f585c14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c073050 0x7f585c148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c163310 con 0x7f585c06bae0 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c14dee0 con 0x7f585c072c70 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.193+0000 7f5862e1a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c14dc20 con 0x7f585c073050 2024-03-16T20:04:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46320/0 (socket says 172.21.15.32:46320) 2024-03-16T20:04:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 -- 172.21.15.32:0/3531853298 learned_addr learned my addr 172.21.15.32:0/3531853298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:04:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f58613b7640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c073050 0x7f585c148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f585bfff640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f585c072c70 0x7f585c14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 -- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c073050 msgr2=0x7f585c148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c073050 0x7f585c148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 -- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f585c072c70 msgr2=0x7f585c14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f585c072c70 0x7f585c14f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.197+0000 7f5860bb6640 1 -- 172.21.15.32:0/3531853298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58500094d0 con 0x7f585c06bae0 2024-03-16T20:04:59.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.201+0000 7f5860bb6640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f585001a160 tx=0x7f585001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.205+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500bc650 con 0x7f585c06bae0 2024-03-16T20:04:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.205+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58500bc7d0 con 0x7f585c06bae0 2024-03-16T20:04:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.205+0000 7f5862e1a640 1 -- 172.21.15.32:0/3531853298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585c14fdd0 con 0x7f585c06bae0 2024-03-16T20:04:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.209+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500bcb80 con 0x7f585c06bae0 2024-03-16T20:04:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.209+0000 7f5862e1a640 1 -- 172.21.15.32:0/3531853298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585c162c50 con 0x7f585c06bae0 2024-03-16T20:04:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.209+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585c163450 con 0x7f585c06bae0 2024-03-16T20:04:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.245+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58500c53f0 con 0x7f585c06bae0 2024-03-16T20:04:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.245+0000 7f58597fa640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 0x7f5840069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:04:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.245+0000 7f585bfff640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 0x7f5840069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:04:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.245+0000 7f585bfff640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 0x7f5840069350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f585c0c71c0 tx=0x7f5848009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:04:59.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.253+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f58500f0580 con 0x7f585c06bae0 2024-03-16T20:04:59.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.613+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f58500ec820 con 0x7f585c06bae0 2024-03-16T20:04:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.805+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f585c163450 con 0x7f585c06bae0 2024-03-16T20:04:59.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f58597fa640 1 -- 172.21.15.32:0/3531853298 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f585c163450 con 0x7f585c06bae0 2024-03-16T20:04:59.898 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:04:59.898 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:04:59.898 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 msgr2=0x7f5840069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 0x7f5840069350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f585c0c71c0 tx=0x7f5848009040 comp rx=0 tx=0).stop 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 msgr2=0x7f585c14e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f585001a160 tx=0x7f585001ad70 comp rx=0 tx=0).stop 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 shutdown_connections 2024-03-16T20:04:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5840066e90 0x7f5840069350 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f585c073050 0x7f585c148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f585c072c70 0x7f585c14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 --2- 172.21.15.32:0/3531853298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f585c06bae0 0x7f585c14e5e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.913+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 >> 172.21.15.32:0/3531853298 conn(0x7f585c06afc0 msgr2=0x7f585c154d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.917+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 shutdown_connections 2024-03-16T20:04:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:04:59.917+0000 7f5846ffd640 1 -- 172.21.15.32:0/3531853298 wait complete. 2024-03-16T20:05:00.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:00.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:01.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:01.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.645+0000 7fac56bd4640 1 Processor -- start 2024-03-16T20:05:01.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.645+0000 7fac56bd4640 1 -- start start 2024-03-16T20:05:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac56bd4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac56bd4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac500c3d80 0x7fac5006f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac56bd4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500c45a0 con 0x7fac5015b760 2024-03-16T20:05:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac56bd4640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac500734e0 con 0x7fac500c3d80 2024-03-16T20:05:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac56bd4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015c250 con 0x7fac500c7ae0 2024-03-16T20:05:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac54970640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fac5015b760 0x7fac5015bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:47712/0 (socket says 172.21.15.32:47712) 2024-03-16T20:05:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac54970640 1 -- 172.21.15.32:0/659156957 learned_addr learned my addr 172.21.15.32:0/659156957 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac4ffff640 1 --2- 172.21.15.32:0/659156957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.649+0000 7fac55171640 1 --2- 172.21.15.32:0/659156957 >> v2:172.21.15.32:3300/0 conn(0x7fac500c3d80 0x7fac5006f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4f7fe640 1 -- 172.21.15.32:0/659156957 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977252637 0 0) 0x7fac500c45a0 con 0x7fac5015b760 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4f7fe640 1 -- 172.21.15.32:0/659156957 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac40003530 con 0x7fac5015b760 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4ffff640 1 -- 172.21.15.32:0/659156957 >> v2:172.21.15.32:3300/0 conn(0x7fac500c3d80 msgr2=0x7fac5006f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4ffff640 1 --2- 172.21.15.32:0/659156957 >> v2:172.21.15.32:3300/0 conn(0x7fac500c3d80 0x7fac5006f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4ffff640 1 -- 172.21.15.32:0/659156957 >> v1:172.21.15.59:6789/0 conn(0x7fac5015b760 legacy=0x7fac5015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.653+0000 7fac4ffff640 1 -- 172.21.15.32:0/659156957 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500c42c0 con 0x7fac500c7ae0 2024-03-16T20:05:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.657+0000 7fac4ffff640 1 --2- 172.21.15.32:0/659156957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c5e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0097c0 tx=0x7fac3c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60bea9e0080c663b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.661+0000 7fac4f7fe640 1 -- 172.21.15.32:0/659156957 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0ac410 con 0x7fac500c7ae0 2024-03-16T20:05:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.661+0000 7fac4f7fe640 1 -- 172.21.15.32:0/659156957 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0ac570 con 0x7fac500c7ae0 2024-03-16T20:05:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.661+0000 7fac4f7fe640 1 -- 172.21.15.32:0/659156957 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0ac940 con 0x7fac500c7ae0 2024-03-16T20:05:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- 172.21.15.32:0/659156957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 msgr2=0x7fac500c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- 172.21.15.32:0/659156957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c5e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0097c0 tx=0x7fac3c01c4c0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- 172.21.15.32:0/659156957 shutdown_connections 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- 172.21.15.32:0/659156957 >> v2:172.21.15.32:3300/0 conn(0x7fac500c3d80 0x7fac5006f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- 172.21.15.32:0/659156957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c5e70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- 172.21.15.32:0/659156957 >> 172.21.15.32:0/659156957 conn(0x7fac500cb470 msgr2=0x7fac500cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- 172.21.15.32:0/659156957 shutdown_connections 2024-03-16T20:05:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- 172.21.15.32:0/659156957 wait complete. 2024-03-16T20:05:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 Processor -- start 2024-03-16T20:05:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- start start 2024-03-16T20:05:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5015b760 0x7fac501e9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5013f5a0 con 0x7fac5015b760 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac50158d50 con 0x7fac500c3d80 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac56bd4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac501f0590 con 0x7fac500c7ae0 2024-03-16T20:05:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac55171640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5015b760 0x7fac501e9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac54970640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac54970640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:34544/0 (socket says 172.21.15.32:34544) 2024-03-16T20:05:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.665+0000 7fac54970640 1 -- 172.21.15.32:0/2775854884 learned_addr learned my addr 172.21.15.32:0/2775854884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac4ffff640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac54970640 1 -- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 msgr2=0x7fac500c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac54970640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac54970640 1 -- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5015b760 msgr2=0x7fac501e9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac54970640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5015b760 0x7fac501e9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.669+0000 7fac54970640 1 -- 172.21.15.32:0/2775854884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c009420 con 0x7fac500c3d80 2024-03-16T20:05:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.673+0000 7fac54970640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fac4400c420 tx=0x7fac4400c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.677+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44005cf0 con 0x7fac500c3d80 2024-03-16T20:05:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.677+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44005e50 con 0x7fac500c3d80 2024-03-16T20:05:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.677+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac501f0870 con 0x7fac500c3d80 2024-03-16T20:05:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.681+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440072d0 con 0x7fac500c3d80 2024-03-16T20:05:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.681+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac501f0e30 con 0x7fac500c3d80 2024-03-16T20:05:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.681+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5006a340 con 0x7fac500c3d80 2024-03-16T20:05:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.713+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fac44007430 con 0x7fac500c3d80 2024-03-16T20:05:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.713+0000 7fac4d7fa640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 0x7fac34069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.717+0000 7fac4ffff640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 0x7fac34069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.717+0000 7fac4ffff640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 0x7fac34069380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0ae000 tx=0x7fac3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:01.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:01.721+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fac44055040 con 0x7fac500c3d80 2024-03-16T20:05:02.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.029+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fac440512e0 con 0x7fac500c3d80 2024-03-16T20:05:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.277+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac500c7560 con 0x7fac500c3d80 2024-03-16T20:05:02.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.301+0000 7fac4d7fa640 1 -- 172.21.15.32:0/2775854884 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fac44017020 con 0x7fac500c3d80 2024-03-16T20:05:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:02.287 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:02.288 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:02.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 msgr2=0x7fac34069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 0x7fac34069380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0ae000 tx=0x7fac3c008040 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 msgr2=0x7fac500ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fac4400c420 tx=0x7fac4400c8e0 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 shutdown_connections 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fac34066ec0 0x7fac34069380 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5015b760 0x7fac501e9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fac500c7ae0 0x7fac500c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 --2- 172.21.15.32:0/2775854884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac500c3d80 0x7fac500ca970 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 >> 172.21.15.32:0/2775854884 conn(0x7fac500cb470 msgr2=0x7fac5015bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 shutdown_connections 2024-03-16T20:05:02.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:02.305+0000 7fac56bd4640 1 -- 172.21.15.32:0/2775854884 wait complete. 2024-03-16T20:05:02.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:02.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:03.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:04.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.033+0000 7f72a908e640 1 Processor -- start 2024-03-16T20:05:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.033+0000 7f72a908e640 1 -- start start 2024-03-16T20:05:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a908e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f72a4151df0 0x7f72a4072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a908e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72a406ba60 0x7f72a40c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a908e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a4073100 con 0x7f72a40c6320 2024-03-16T20:05:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a908e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a40c6770 con 0x7f72a406ba60 2024-03-16T20:05:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a908e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a4070890 con 0x7f72a4151df0 2024-03-16T20:05:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72a406ba60 0x7f72a40c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72a406ba60 0x7f72a40c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46354/0 (socket says 172.21.15.32:46354) 2024-03-16T20:05:04.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a259c640 1 -- 172.21.15.32:0/3351994356 learned_addr learned my addr 172.21.15.32:0/3351994356 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:04.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.037+0000 7f72a2d9d640 1 --2- 172.21.15.32:0/3351994356 >> v2:172.21.15.59:3300/0 conn(0x7f72a4151df0 0x7f72a4072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.041+0000 7f72a259c640 1 -- 172.21.15.32:0/3351994356 >> v2:172.21.15.59:3300/0 conn(0x7f72a4151df0 msgr2=0x7f72a4072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:04.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.041+0000 7f72a259c640 1 --2- 172.21.15.32:0/3351994356 >> v2:172.21.15.59:3300/0 conn(0x7f72a4151df0 0x7f72a4072bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.041+0000 7f72a259c640 1 -- 172.21.15.32:0/3351994356 >> v1:172.21.15.32:6789/0 conn(0x7f72a40c6320 legacy=0x7f72a4158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.041+0000 7f72a259c640 1 -- 172.21.15.32:0/3351994356 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4073370 con 0x7f72a406ba60 2024-03-16T20:05:04.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.049+0000 7f72a259c640 1 --2- 172.21.15.32:0/3351994356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a406ba60 0x7f72a40c5de0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f72a406b630 tx=0x7f729801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd1e61d87700a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:04.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a1d9b640 1 -- 172.21.15.32:0/3351994356 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980a8a10 con 0x7f72a406ba60 2024-03-16T20:05:04.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a1d9b640 1 -- 172.21.15.32:0/3351994356 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980a8b70 con 0x7f72a406ba60 2024-03-16T20:05:04.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a1d9b640 1 -- 172.21.15.32:0/3351994356 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980b0440 con 0x7f72a406ba60 2024-03-16T20:05:04.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 -- 172.21.15.32:0/3351994356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a406ba60 msgr2=0x7f72a40c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 --2- 172.21.15.32:0/3351994356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a406ba60 0x7f72a40c5de0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f72a406b630 tx=0x7f729801ed10 comp rx=0 tx=0).stop 2024-03-16T20:05:04.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 -- 172.21.15.32:0/3351994356 shutdown_connections 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 --2- 172.21.15.32:0/3351994356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a406ba60 0x7f72a40c5de0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 --2- 172.21.15.32:0/3351994356 >> v2:172.21.15.59:3300/0 conn(0x7f72a4151df0 0x7f72a4072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 -- 172.21.15.32:0/3351994356 >> 172.21.15.32:0/3351994356 conn(0x7f72a406c0d0 msgr2=0x7f72a406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 -- 172.21.15.32:0/3351994356 shutdown_connections 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.053+0000 7f72a908e640 1 -- 172.21.15.32:0/3351994356 wait complete. 2024-03-16T20:05:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 Processor -- start 2024-03-16T20:05:04.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 -- start start 2024-03-16T20:05:04.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a406ba60 0x7f72a41f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72a40c6320 0x7f72a414f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a41431f0 con 0x7f72a4151df0 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c4640 con 0x7f72a406ba60 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a908e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c4900 con 0x7f72a40c6320 2024-03-16T20:05:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a359e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46356/0 (socket says 172.21.15.32:46356) 2024-03-16T20:05:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a359e640 1 -- 172.21.15.32:0/1681132664 learned_addr learned my addr 172.21.15.32:0/1681132664 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a259c640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72a40c6320 0x7f72a414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.057+0000 7f72a2d9d640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a406ba60 0x7f72a41f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.061+0000 7f72a359e640 1 -- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72a40c6320 msgr2=0x7f72a414f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.061+0000 7f72a359e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72a40c6320 0x7f72a414f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.061+0000 7f72a359e640 1 -- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a406ba60 msgr2=0x7f72a41f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.061+0000 7f72a359e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a406ba60 0x7f72a41f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.061+0000 7f72a359e640 1 -- 172.21.15.32:0/1681132664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980094d0 con 0x7f72a4151df0 2024-03-16T20:05:04.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.065+0000 7f72a359e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c770 tx=0x7f729c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:04.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c003b80 con 0x7f72a4151df0 2024-03-16T20:05:04.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c003ce0 con 0x7f72a4151df0 2024-03-16T20:05:04.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c00aa90 con 0x7f72a4151df0 2024-03-16T20:05:04.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a40c4be0 con 0x7f72a4151df0 2024-03-16T20:05:04.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a40c4ec0 con 0x7f72a4151df0 2024-03-16T20:05:04.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.069+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4068000 con 0x7f72a4151df0 2024-03-16T20:05:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.101+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f729c00abf0 con 0x7f72a4151df0 2024-03-16T20:05:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.101+0000 7f728b7fe640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.101+0000 7f72a2d9d640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.101+0000 7f72a2d9d640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7294009b60 tx=0x7f7294009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.393+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f729c055e00 con 0x7f72a4151df0 2024-03-16T20:05:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.393+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f729c0520a0 con 0x7f72a4151df0 2024-03-16T20:05:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.593+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72a4143660 con 0x7f72a4151df0 2024-03-16T20:05:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.609+0000 7f728b7fe640 1 -- 172.21.15.32:0/1681132664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f729c055760 con 0x7f72a4151df0 2024-03-16T20:05:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:04.594 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.609+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 msgr2=0x7f7290069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.609+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7294009b60 tx=0x7f7294009190 comp rx=0 tx=0).stop 2024-03-16T20:05:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.609+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 msgr2=0x7f72a41439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.609+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c770 tx=0x7f729c00cc30 comp rx=0 tx=0).stop 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 shutdown_connections 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7290066e90 0x7f7290069350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72a4151df0 0x7f72a41439b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f72a40c6320 0x7f72a414f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 --2- 172.21.15.32:0/1681132664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a406ba60 0x7f72a41f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 >> 172.21.15.32:0/1681132664 conn(0x7f72a406c0d0 msgr2=0x7f72a4150730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 shutdown_connections 2024-03-16T20:05:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:04.613+0000 7f72a908e640 1 -- 172.21.15.32:0/1681132664 wait complete. 2024-03-16T20:05:05.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:05.217 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:06.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 Processor -- start 2024-03-16T20:05:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 -- start start 2024-03-16T20:05:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa770153fa0 0x7fa770070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700c8590 con 0x7fa7700724a0 2024-03-16T20:05:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa770073590 con 0x7fa7700c7470 2024-03-16T20:05:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa77672e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7700c3f00 con 0x7fa770153fa0 2024-03-16T20:05:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa774ccb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7700724a0 0x7fa770164770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41788/0 (socket says 172.21.15.32:41788) 2024-03-16T20:05:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.341+0000 7fa774ccb640 1 -- 172.21.15.32:0/1235601093 learned_addr learned my addr 172.21.15.32:0/1235601093 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.345+0000 7fa76ffff640 1 --2- 172.21.15.32:0/1235601093 >> v2:172.21.15.59:3300/0 conn(0x7fa770153fa0 0x7fa770070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.345+0000 7fa76effd640 1 -- 172.21.15.32:0/1235601093 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446629863 0 0) 0x7fa7700c8590 con 0x7fa7700724a0 2024-03-16T20:05:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.345+0000 7fa76effd640 1 -- 172.21.15.32:0/1235601093 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa758003530 con 0x7fa7700724a0 2024-03-16T20:05:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.349+0000 7fa76ffff640 1 -- 172.21.15.32:0/1235601093 >> v1:172.21.15.59:6789/0 conn(0x7fa7700c7470 legacy=0x7fa77015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.349+0000 7fa76ffff640 1 -- 172.21.15.32:0/1235601093 >> v1:172.21.15.32:6789/0 conn(0x7fa7700724a0 legacy=0x7fa770164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.349+0000 7fa76ffff640 1 -- 172.21.15.32:0/1235601093 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7700c78c0 con 0x7fa770153fa0 2024-03-16T20:05:06.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.353+0000 7fa76ffff640 1 --2- 172.21.15.32:0/1235601093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa770153fa0 0x7fa770070840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa7600097c0 tx=0x7fa76001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc98ef4c06ac82bc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.361+0000 7fa76effd640 1 -- 172.21.15.32:0/1235601093 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600a8950 con 0x7fa770153fa0 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.361+0000 7fa76effd640 1 -- 172.21.15.32:0/1235601093 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600a8ab0 con 0x7fa770153fa0 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.361+0000 7fa76effd640 1 -- 172.21.15.32:0/1235601093 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600a8e40 con 0x7fa770153fa0 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- 172.21.15.32:0/1235601093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa770153fa0 msgr2=0x7fa770070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 --2- 172.21.15.32:0/1235601093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa770153fa0 0x7fa770070840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa7600097c0 tx=0x7fa76001ec50 comp rx=0 tx=0).stop 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- 172.21.15.32:0/1235601093 shutdown_connections 2024-03-16T20:05:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 --2- 172.21.15.32:0/1235601093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa770153fa0 0x7fa770070840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- 172.21.15.32:0/1235601093 >> 172.21.15.32:0/1235601093 conn(0x7fa77006c0d0 msgr2=0x7fa77006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- 172.21.15.32:0/1235601093 shutdown_connections 2024-03-16T20:05:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- 172.21.15.32:0/1235601093 wait complete. 2024-03-16T20:05:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 Processor -- start 2024-03-16T20:05:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- start start 2024-03-16T20:05:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7700724a0 0x7fa77015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7700c7470 0x7fa7701546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700c6890 con 0x7fa770153fa0 2024-03-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77015cf20 con 0x7fa7700724a0 2024-03-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa77672e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77015be00 con 0x7fa7700c7470 2024-03-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa774ccb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa76f7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7700c7470 0x7fa7701546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa774ccb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46362/0 (socket says 172.21.15.32:46362) 2024-03-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa774ccb640 1 -- 172.21.15.32:0/3059620876 learned_addr learned my addr 172.21.15.32:0/3059620876 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.365+0000 7fa76ffff640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7700724a0 0x7fa77015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.373+0000 7fa774ccb640 1 -- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7700c7470 msgr2=0x7fa7701546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.373+0000 7fa774ccb640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7700c7470 0x7fa7701546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.373+0000 7fa774ccb640 1 -- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7700724a0 msgr2=0x7fa77015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.373+0000 7fa774ccb640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7700724a0 0x7fa77015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.373+0000 7fa774ccb640 1 -- 172.21.15.32:0/3059620876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760009420 con 0x7fa770153fa0 2024-03-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.377+0000 7fa774ccb640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa76800caa0 tx=0x7fa76800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76800ac00 con 0x7fa770153fa0 2024-03-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76800ad60 con 0x7fa770153fa0 2024-03-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77015dfd0 con 0x7fa770153fa0 2024-03-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768012620 con 0x7fa770153fa0 2024-03-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa770150f10 con 0x7fa770153fa0 2024-03-16T20:05:06.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.381+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77015d0d0 con 0x7fa770153fa0 2024-03-16T20:05:06.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.413+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa768007070 con 0x7fa770153fa0 2024-03-16T20:05:06.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.413+0000 7fa76cff9640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 0x7fa75c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:06.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.413+0000 7fa76ffff640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 0x7fa75c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:06.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.413+0000 7fa76ffff640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 0x7fa75c069350 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa7600b2000 tx=0x7fa7600aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:06.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.417+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa768055810 con 0x7fa770153fa0 2024-03-16T20:05:06.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.709+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa768051ab0 con 0x7fa770153fa0 2024-03-16T20:05:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.917+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7700c7f00 con 0x7fa770153fa0 2024-03-16T20:05:06.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.973+0000 7fa76cff9640 1 -- 172.21.15.32:0/3059620876 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa768017090 con 0x7fa770153fa0 2024-03-16T20:05:06.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:06.960 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:06.960 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 msgr2=0x7fa75c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 0x7fa75c069350 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa7600b2000 tx=0x7fa7600aa070 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 msgr2=0x7fa770154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa76800caa0 tx=0x7fa76800cf60 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 shutdown_connections 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa75c066e90 0x7fa75c069350 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa770153fa0 0x7fa770154c20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7700c7470 0x7fa7701546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 --2- 172.21.15.32:0/3059620876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7700724a0 0x7fa77015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 >> 172.21.15.32:0/3059620876 conn(0x7fa77006c0d0 msgr2=0x7fa770149160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 shutdown_connections 2024-03-16T20:05:06.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:06.977+0000 7fa77672e640 1 -- 172.21.15.32:0/3059620876 wait complete. 2024-03-16T20:05:07.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:07.590 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:08.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 Processor -- start 2024-03-16T20:05:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 -- start start 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 0x7fb6840c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb68415b2a0 0x7fb68415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb684158c80 con 0x7fb684151ef0 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6840b0ad0 con 0x7fb68415b2a0 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb68b5b7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb684157cb0 con 0x7fb6840c81e0 2024-03-16T20:05:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb688b52640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 0x7fb6840c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.713+0000 7fb689b54640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb68415b2a0 0x7fb68415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb688b52640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 0x7fb6840c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:34588/0 (socket says 172.21.15.32:34588) 2024-03-16T20:05:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb688b52640 1 -- 172.21.15.32:0/3380493158 learned_addr learned my addr 172.21.15.32:0/3380493158 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:08.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb689b54640 1 -- 172.21.15.32:0/3380493158 >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 msgr2=0x7fb6840c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:08.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb689b54640 1 --2- 172.21.15.32:0/3380493158 >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 0x7fb6840c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb689b54640 1 -- 172.21.15.32:0/3380493158 >> v1:172.21.15.32:6789/0 conn(0x7fb684151ef0 legacy=0x7fb6840c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.717+0000 7fb689b54640 1 -- 172.21.15.32:0/3380493158 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68415bfc0 con 0x7fb68415b2a0 2024-03-16T20:05:08.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.725+0000 7fb689b54640 1 --2- 172.21.15.32:0/3380493158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb68415b6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb6840cad10 tx=0x7fb67c01f0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0416791caf07f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb67bfff640 1 -- 172.21.15.32:0/3380493158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0b0440 con 0x7fb68415b2a0 2024-03-16T20:05:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb67bfff640 1 -- 172.21.15.32:0/3380493158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c0b05a0 con 0x7fb68415b2a0 2024-03-16T20:05:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb67bfff640 1 -- 172.21.15.32:0/3380493158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0b0930 con 0x7fb68415b2a0 2024-03-16T20:05:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3380493158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 msgr2=0x7fb68415b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3380493158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb68415b6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb6840cad10 tx=0x7fb67c01f0e0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3380493158 shutdown_connections 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3380493158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb68415b6a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3380493158 >> v2:172.21.15.59:3300/0 conn(0x7fb6840c81e0 0x7fb6840c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3380493158 >> 172.21.15.32:0/3380493158 conn(0x7fb684070f90 msgr2=0x7fb6840713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3380493158 shutdown_connections 2024-03-16T20:05:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.749+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3380493158 wait complete. 2024-03-16T20:05:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 Processor -- start 2024-03-16T20:05:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 -- start start 2024-03-16T20:05:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6840c81e0 0x7fb6841f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb684151ef0 0x7fb6841ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840b20b0 con 0x7fb68415b2a0 2024-03-16T20:05:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841f1d40 con 0x7fb6840c81e0 2024-03-16T20:05:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb68b5b7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841f14a0 con 0x7fb684151ef0 2024-03-16T20:05:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb689b54640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb689353640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6840c81e0 0x7fb6841f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb689b54640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46370/0 (socket says 172.21.15.32:46370) 2024-03-16T20:05:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb689b54640 1 -- 172.21.15.32:0/3734496004 learned_addr learned my addr 172.21.15.32:0/3734496004 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.753+0000 7fb688b52640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb684151ef0 0x7fb6841ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.757+0000 7fb689b54640 1 -- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb684151ef0 msgr2=0x7fb6841ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:08.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.757+0000 7fb689b54640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb684151ef0 0x7fb6841ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.757+0000 7fb689b54640 1 -- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6840c81e0 msgr2=0x7fb6841f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:08.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.757+0000 7fb689b54640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6840c81e0 0x7fb6841f0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:08.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.757+0000 7fb689b54640 1 -- 172.21.15.32:0/3734496004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009a70 con 0x7fb68415b2a0 2024-03-16T20:05:08.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.761+0000 7fb689b54640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb67c01cd10 tx=0x7fb67c01be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0b0440 con 0x7fb68415b2a0 2024-03-16T20:05:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c0ba610 con 0x7fb68415b2a0 2024-03-16T20:05:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6841f1780 con 0x7fb68415b2a0 2024-03-16T20:05:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0ba980 con 0x7fb68415b2a0 2024-03-16T20:05:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6841eda60 con 0x7fb68415b2a0 2024-03-16T20:05:08.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.765+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6841ed8c0 con 0x7fb68415b2a0 2024-03-16T20:05:08.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.789+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb67c0b7050 con 0x7fb68415b2a0 2024-03-16T20:05:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.789+0000 7fb679ffb640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 0x7fb668069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.789+0000 7fb689353640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 0x7fb668069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:08.789+0000 7fb689353640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 0x7fb668069300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb68414aa60 tx=0x7fb680009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.093+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb67c0f41b0 con 0x7fb68415b2a0 2024-03-16T20:05:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.093+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb67c0f0450 con 0x7fb68415b2a0 2024-03-16T20:05:09.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.309+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6841ed8c0 con 0x7fb68415b2a0 2024-03-16T20:05:09.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.325+0000 7fb679ffb640 1 -- 172.21.15.32:0/3734496004 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fb6841ed8c0 con 0x7fb68415b2a0 2024-03-16T20:05:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:09.310 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:09.310 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:09.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 msgr2=0x7fb668069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:09.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 0x7fb668069300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb68414aa60 tx=0x7fb680009040 comp rx=0 tx=0).stop 2024-03-16T20:05:09.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 msgr2=0x7fb6840b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:09.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb67c01cd10 tx=0x7fb67c01be80 comp rx=0 tx=0).stop 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 shutdown_connections 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb668066e40 0x7fb668069300 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68415b2a0 0x7fb6840b13a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb684151ef0 0x7fb6841ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 --2- 172.21.15.32:0/3734496004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6840c81e0 0x7fb6841f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 >> 172.21.15.32:0/3734496004 conn(0x7fb684070f90 msgr2=0x7fb684155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 shutdown_connections 2024-03-16T20:05:09.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:09.329+0000 7fb68b5b7640 1 -- 172.21.15.32:0/3734496004 wait complete. 2024-03-16T20:05:09.544 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:53.632 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.562 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:53.650 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.590 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:53.614 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.594 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:53.619 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.595 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:53.619 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.599 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:53.624 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.640 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:53.728 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.673 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:09:53.698 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.687 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:09:53.711 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:05:09.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:09.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:10.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 Processor -- start 2024-03-16T20:05:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 -- start start 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c4072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c406b3f0 con 0x7f97c40c9de0 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c406b6b0 con 0x7f97c4154030 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.065+0000 7f97ca98d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c406b990 con 0x7f97c4154410 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.069+0000 7f97c37fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f97c40c9de0 0x7f97c40cc620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38920/0 (socket says 172.21.15.32:38920) 2024-03-16T20:05:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.069+0000 7f97c37fe640 1 -- 172.21.15.32:0/998377020 learned_addr learned my addr 172.21.15.32:0/998377020 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.069+0000 7f97c8f2a640 1 --2- 172.21.15.32:0/998377020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c4072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:11.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269885621 0 0) 0x7f97c406b3f0 con 0x7f97c40c9de0 2024-03-16T20:05:11.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac003530 con 0x7f97c40c9de0 2024-03-16T20:05:11.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348285967 0 0) 0x7f97c406b6b0 con 0x7f97c4154030 2024-03-16T20:05:11.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c406b3f0 con 0x7f97c4154030 2024-03-16T20:05:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c8f2a640 1 -- 172.21.15.32:0/998377020 >> v1:172.21.15.59:6789/0 conn(0x7f97c4154030 legacy=0x7f97c4159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c8f2a640 1 -- 172.21.15.32:0/998377020 >> v1:172.21.15.32:6789/0 conn(0x7f97c40c9de0 legacy=0x7f97c40cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.073+0000 7f97c8f2a640 1 -- 172.21.15.32:0/998377020 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40c9300 con 0x7f97c4154410 2024-03-16T20:05:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.081+0000 7f97c8f2a640 1 --2- 172.21.15.32:0/998377020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c4072ec0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009dc0 tx=0x7f97bc01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a72a4100f5398b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0af600 con 0x7f97c4154410 2024-03-16T20:05:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc0ab050 con 0x7f97c4154410 2024-03-16T20:05:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97c2ffd640 1 -- 172.21.15.32:0/998377020 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0af980 con 0x7f97c4154410 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 -- 172.21.15.32:0/998377020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 msgr2=0x7f97c4072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 --2- 172.21.15.32:0/998377020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c4072ec0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009dc0 tx=0x7f97bc01f170 comp rx=0 tx=0).stop 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 -- 172.21.15.32:0/998377020 shutdown_connections 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 --2- 172.21.15.32:0/998377020 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c4072ec0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 -- 172.21.15.32:0/998377020 >> 172.21.15.32:0/998377020 conn(0x7f97c406e6c0 msgr2=0x7f97c406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 -- 172.21.15.32:0/998377020 shutdown_connections 2024-03-16T20:05:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.089+0000 7f97ca98d640 1 -- 172.21.15.32:0/998377020 wait complete. 2024-03-16T20:05:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 Processor -- start 2024-03-16T20:05:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 -- start start 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40c9de0 0x7f97c4156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c41efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4155d60 con 0x7f97c4154030 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4159710 con 0x7f97c40c9de0 2024-03-16T20:05:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97ca98d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4157ec0 con 0x7f97c4154410 2024-03-16T20:05:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97c37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97c3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40c9de0 0x7f97c4156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97c37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60294/0 (socket says 172.21.15.32:60294) 2024-03-16T20:05:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.093+0000 7f97c37fe640 1 -- 172.21.15.32:0/3441130315 learned_addr learned my addr 172.21.15.32:0/3441130315 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c8f2a640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c41efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c37fe640 1 -- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 msgr2=0x7f97c41efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c37fe640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c41efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c37fe640 1 -- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40c9de0 msgr2=0x7f97c4156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c37fe640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40c9de0 0x7f97c4156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.097+0000 7f97c37fe640 1 -- 172.21.15.32:0/3441130315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc009a70 con 0x7f97c4154030 2024-03-16T20:05:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.101+0000 7f97c37fe640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c500 tx=0x7f97b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.101+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8005cf0 con 0x7f97c4154030 2024-03-16T20:05:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.101+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c41576e0 con 0x7f97c4154030 2024-03-16T20:05:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.101+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c4159c40 con 0x7f97c4154030 2024-03-16T20:05:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.105+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c4068000 con 0x7f97c4154030 2024-03-16T20:05:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.141+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8005e50 con 0x7f97c4154030 2024-03-16T20:05:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.141+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b80072f0 con 0x7f97c4154030 2024-03-16T20:05:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.145+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f97b8007490 con 0x7f97c4154030 2024-03-16T20:05:11.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.145+0000 7f97c0ff9640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 0x7f97b0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:11.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.145+0000 7f97c3fff640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 0x7f97b0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:11.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.145+0000 7f97c3fff640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 0x7f97b0069330 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97b40011f0 tx=0x7f97b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:11.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.417+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f97b8007740 con 0x7f97c4154030 2024-03-16T20:05:11.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.421+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f97b80516a0 con 0x7f97c4154030 2024-03-16T20:05:11.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.705+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c40c8bb0 con 0x7f97c4154030 2024-03-16T20:05:11.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.793+0000 7f97c0ff9640 1 -- 172.21.15.32:0/3441130315 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f97b8021090 con 0x7f97c4154030 2024-03-16T20:05:11.777 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:11.778 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 msgr2=0x7f97b0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 0x7f97b0069330 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97b40011f0 tx=0x7f97b4009040 comp rx=0 tx=0).stop 2024-03-16T20:05:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 msgr2=0x7f97c414e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c500 tx=0x7f97b800c9c0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 shutdown_connections 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f97b0066e70 0x7f97b0069330 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97c4154410 0x7f97c41efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c4154030 0x7f97c414e280 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 --2- 172.21.15.32:0/3441130315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40c9de0 0x7f97c4156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 >> 172.21.15.32:0/3441130315 conn(0x7f97c406e6c0 msgr2=0x7f97c406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 shutdown_connections 2024-03-16T20:05:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:11.797+0000 7f97ca98d640 1 -- 172.21.15.32:0/3441130315 wait complete. 2024-03-16T20:05:12.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:05:12.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:05:12.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:12.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:13.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 Processor -- start 2024-03-16T20:05:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 -- start start 2024-03-16T20:05:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440015b9b0 0x7f44000c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400158c80 con 0x7f4400151ef0 2024-03-16T20:05:13.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440015bfc0 con 0x7f44000c81e0 2024-03-16T20:05:13.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.537+0000 7f44068bc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000b0ad0 con 0x7f440015b9b0 2024-03-16T20:05:13.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.541+0000 7f43ff7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f44000c81e0 0x7f440015b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:32840/0 (socket says 172.21.15.32:32840) 2024-03-16T20:05:13.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.541+0000 7f43ff7fe640 1 -- 172.21.15.32:0/4100144156 learned_addr learned my addr 172.21.15.32:0/4100144156 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.545+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680074434 0 0) 0x7f440015bfc0 con 0x7f44000c81e0 2024-03-16T20:05:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.545+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8003530 con 0x7f44000c81e0 2024-03-16T20:05:13.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.1 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774026782 0 0) 0x7f43e8003530 con 0x7f44000c81e0 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 >> v2:172.21.15.32:3300/0 conn(0x7f440015b9b0 msgr2=0x7f44000c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 --2- 172.21.15.32:0/4100144156 >> v2:172.21.15.32:3300/0 conn(0x7f440015b9b0 0x7f44000c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 >> v1:172.21.15.32:6789/0 conn(0x7f4400151ef0 legacy=0x7f44000c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400157ec0 con 0x7f44000c81e0 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.1 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f43f4001680 con 0x7f44000c81e0 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 >> v1:172.21.15.59:6789/0 conn(0x7f44000c81e0 legacy=0x7f440015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43e8003980 0x7f43e8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43e80047d0 0x7f43e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f43e800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b0ad0 con 0x7f43e80052b0 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e800c1a0 con 0x7f43e80047d0 2024-03-16T20:05:13.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.553+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e800c360 con 0x7f43e8003980 2024-03-16T20:05:13.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.557+0000 7f43fffff640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43e8003980 0x7f43e8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.557+0000 7f43ff7fe640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43e80047d0 0x7f43e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.561+0000 7f4404e59640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f43e800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.565+0000 7f4404e59640 1 -- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43e8003980 msgr2=0x7f43e8003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:13.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.565+0000 7f4404e59640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43e8003980 0x7f43e8003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.565+0000 7f4404e59640 1 -- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43e80047d0 msgr2=0x7f43e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:13.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.565+0000 7f4404e59640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43e80047d0 0x7f43e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.565+0000 7f4404e59640 1 -- 172.21.15.32:0/4100144156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8003530 con 0x7f43e80052b0 2024-03-16T20:05:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.569+0000 7f4404e59640 1 -- 172.21.15.32:0/4100144156 reap_dead start 2024-03-16T20:05:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.569+0000 7f4404e59640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f43e800b990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009c70 tx=0x7f43f801cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0563795b77529c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:13.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80a5c60 con 0x7f43e80052b0 2024-03-16T20:05:13.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f80a5dc0 con 0x7f43e80052b0 2024-03-16T20:05:13.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f43feffd640 1 -- 172.21.15.32:0/4100144156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8003bc0 con 0x7f43e80052b0 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 -- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 msgr2=0x7f43e800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f43e800b990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009c70 tx=0x7f43f801cc50 comp rx=0 tx=0).stop 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 -- 172.21.15.32:0/4100144156 shutdown_connections 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 --2- 172.21.15.32:0/4100144156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f43e800b990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 -- 172.21.15.32:0/4100144156 >> 172.21.15.32:0/4100144156 conn(0x7f4400070f90 msgr2=0x7f44000713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 -- 172.21.15.32:0/4100144156 shutdown_connections 2024-03-16T20:05:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.573+0000 7f44068bc640 1 -- 172.21.15.32:0/4100144156 wait complete. 2024-03-16T20:05:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 Processor -- start 2024-03-16T20:05:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 -- start start 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f4400156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f440015a9a0 0x7f440015ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 0x7f44001f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440015b780 con 0x7f43e80052b0 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44001edca0 con 0x7f44001f1070 2024-03-16T20:05:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f44068bc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44001edf80 con 0x7f440015a9a0 2024-03-16T20:05:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f4404e59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 0x7f44001f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f43ff7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f440015a9a0 0x7f440015ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f4404e59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 0x7f44001f14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:52030/0 (socket says 172.21.15.32:52030) 2024-03-16T20:05:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.577+0000 7f4404e59640 1 -- 172.21.15.32:0/3697177201 learned_addr learned my addr 172.21.15.32:0/3697177201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:13.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.581+0000 7f43fffff640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f4400156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 -- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f440015a9a0 msgr2=0x7f440015ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f440015a9a0 0x7f440015ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 -- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 msgr2=0x7f44001f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 0x7f44001f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 -- 172.21.15.32:0/3697177201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44001ee260 con 0x7f43e80052b0 2024-03-16T20:05:13.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.585+0000 7f43fffff640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f4400156d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f43f000f4d0 tx=0x7f43f000f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.589+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0005320 con 0x7f43e80052b0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.589+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00054a0 con 0x7f43e80052b0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.589+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0007290 con 0x7f43e80052b0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.589+0000 7f44068bc640 1 -- 172.21.15.32:0/3697177201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44001ee540 con 0x7f43e80052b0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.593+0000 7f44068bc640 1 -- 172.21.15.32:0/3697177201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44001572c0 con 0x7f43e80052b0 2024-03-16T20:05:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.593+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d80052a0 con 0x7f43e80052b0 2024-03-16T20:05:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.877+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f43f00073f0 con 0x7f43e80052b0 2024-03-16T20:05:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.877+0000 7f43fcff9640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 0x7f43ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.877+0000 7f43ff7fe640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 0x7f43ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.877+0000 7f43ff7fe640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 0x7f43ec069350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43f4001c70 tx=0x7f43f4001b00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:13.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.901+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f43f00076a0 con 0x7f43e80052b0 2024-03-16T20:05:13.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:13.901+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f43f0051510 con 0x7f43e80052b0 2024-03-16T20:05:14.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.113+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43d8005900 con 0x7f43e80052b0 2024-03-16T20:05:14.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.201+0000 7f43fcff9640 1 -- 172.21.15.32:0/3697177201 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f43d8005900 con 0x7f43e80052b0 2024-03-16T20:05:14.184 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:14.184 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:14.185 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:14.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 msgr2=0x7f43ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:14.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 0x7f43ec069350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43f4001c70 tx=0x7f43f4001b00 comp rx=0 tx=0).stop 2024-03-16T20:05:14.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 msgr2=0x7f4400156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:14.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f4400156d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f43f000f4d0 tx=0x7f43f000f8d0 comp rx=0 tx=0).stop 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 shutdown_connections 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f43ec066e90 0x7f43ec069350 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44001f1070 0x7f44001f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f440015a9a0 0x7f440015ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 --2- 172.21.15.32:0/3697177201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43e80052b0 0x7f4400156d80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 >> 172.21.15.32:0/3697177201 conn(0x7f4400070f90 msgr2=0x7f440014f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 shutdown_connections 2024-03-16T20:05:14.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:14.205+0000 7f43e67fc640 1 -- 172.21.15.32:0/3697177201 wait complete. 2024-03-16T20:05:14.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:14.813 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:15.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.941+0000 7f4979e1e640 1 Processor -- start 2024-03-16T20:05:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.941+0000 7f4979e1e640 1 -- start start 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4979e1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4974150c40 0x7f4974073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4979e1e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49740725e0 0x7f4974070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4979e1e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4974151020 con 0x7f49740708a0 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4979e1e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497415a000 con 0x7f49740725e0 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4979e1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4974072da0 con 0x7f4974150c40 2024-03-16T20:05:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4972ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49740725e0 0x7f4974070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f49737fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4974150c40 0x7f4974073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4972ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49740725e0 0x7f4974070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60328/0 (socket says 172.21.15.32:60328) 2024-03-16T20:05:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.945+0000 7f4972ffd640 1 -- 172.21.15.32:0/527552407 learned_addr learned my addr 172.21.15.32:0/527552407 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.949+0000 7f4972ffd640 1 -- 172.21.15.32:0/527552407 >> v2:172.21.15.59:3300/0 conn(0x7f4974150c40 msgr2=0x7f4974073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.949+0000 7f4972ffd640 1 --2- 172.21.15.32:0/527552407 >> v2:172.21.15.59:3300/0 conn(0x7f4974150c40 0x7f4974073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.949+0000 7f4972ffd640 1 -- 172.21.15.32:0/527552407 >> v1:172.21.15.59:6789/0 conn(0x7f49740708a0 legacy=0x7f497415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.949+0000 7f4972ffd640 1 -- 172.21.15.32:0/527552407 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497415e310 con 0x7f49740725e0 2024-03-16T20:05:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.949+0000 7f4972ffd640 1 --2- 172.21.15.32:0/527552407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740725e0 0x7f4974070360 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f497415e500 tx=0x7f496801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9f2680cab7928a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f49727fc640 1 -- 172.21.15.32:0/527552407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680a7070 con 0x7f49740725e0 2024-03-16T20:05:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f49727fc640 1 -- 172.21.15.32:0/527552407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49680ab430 con 0x7f49740725e0 2024-03-16T20:05:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f49727fc640 1 -- 172.21.15.32:0/527552407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680ab7c0 con 0x7f49740725e0 2024-03-16T20:05:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- 172.21.15.32:0/527552407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740725e0 msgr2=0x7f4974070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- 172.21.15.32:0/527552407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740725e0 0x7f4974070360 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f497415e500 tx=0x7f496801c520 comp rx=0 tx=0).stop 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- 172.21.15.32:0/527552407 shutdown_connections 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- 172.21.15.32:0/527552407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740725e0 0x7f4974070360 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- 172.21.15.32:0/527552407 >> v2:172.21.15.59:3300/0 conn(0x7f4974150c40 0x7f4974073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- 172.21.15.32:0/527552407 >> 172.21.15.32:0/527552407 conn(0x7f497406c0d0 msgr2=0x7f497406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- 172.21.15.32:0/527552407 shutdown_connections 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- 172.21.15.32:0/527552407 wait complete. 2024-03-16T20:05:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 Processor -- start 2024-03-16T20:05:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- start start 2024-03-16T20:05:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49740708a0 0x7f49740c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740725e0 0x7f497413d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497413ba20 con 0x7f4974150c40 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497413f060 con 0x7f49740725e0 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4979e1e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497415c9e0 con 0x7f49740708a0 2024-03-16T20:05:15.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.957+0000 7f4973fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.961+0000 7f4973fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60344/0 (socket says 172.21.15.32:60344) 2024-03-16T20:05:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.961+0000 7f4972ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740725e0 0x7f497413d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.961+0000 7f4973fff640 1 -- 172.21.15.32:0/2050395180 learned_addr learned my addr 172.21.15.32:0/2050395180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.961+0000 7f49737fe640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49740708a0 0x7f49740c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.965+0000 7f4973fff640 1 -- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49740708a0 msgr2=0x7f49740c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.965+0000 7f4973fff640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49740708a0 0x7f49740c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.965+0000 7f4973fff640 1 -- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740725e0 msgr2=0x7f497413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.965+0000 7f4973fff640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740725e0 0x7f497413d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.965+0000 7f4973fff640 1 -- 172.21.15.32:0/2050395180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49680094d0 con 0x7f4974150c40 2024-03-16T20:05:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.969+0000 7f4973fff640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f496c00c730 tx=0x7f496c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c003b80 con 0x7f4974150c40 2024-03-16T20:05:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c003ce0 con 0x7f4974150c40 2024-03-16T20:05:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497415d560 con 0x7f4974150c40 2024-03-16T20:05:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c00aa30 con 0x7f4974150c40 2024-03-16T20:05:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497413e050 con 0x7f4974150c40 2024-03-16T20:05:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:15.973+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4974068000 con 0x7f4974150c40 2024-03-16T20:05:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.009+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f496c00ab90 con 0x7f4974150c40 2024-03-16T20:05:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.009+0000 7f495bfff640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 0x7f4960069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.009+0000 7f49737fe640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 0x7f4960069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:15.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.009+0000 7f49737fe640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 0x7f4960069300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4964007a60 tx=0x7f4964008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:15.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.013+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f496c00ae40 con 0x7f4974150c40 2024-03-16T20:05:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.309+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f496c052100 con 0x7f4974150c40 2024-03-16T20:05:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.505+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f497413f1a0 con 0x7f4974150c40 2024-03-16T20:05:16.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.525+0000 7f495bfff640 1 -- 172.21.15.32:0/2050395180 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f496c0557c0 con 0x7f4974150c40 2024-03-16T20:05:16.508 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:16.508 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:16.508 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 msgr2=0x7f4960069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 0x7f4960069300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4964007a60 tx=0x7f4964008040 comp rx=0 tx=0).stop 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 msgr2=0x7f497413d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f496c00c730 tx=0x7f496c00cbf0 comp rx=0 tx=0).stop 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 shutdown_connections 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4960066e40 0x7f4960069300 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4974150c40 0x7f497413d830 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740725e0 0x7f497413d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:16.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 --2- 172.21.15.32:0/2050395180 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49740708a0 0x7f49740c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:16.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 >> 172.21.15.32:0/2050395180 conn(0x7f497406c0d0 msgr2=0x7f497414ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:16.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 shutdown_connections 2024-03-16T20:05:16.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:16.529+0000 7f4979e1e640 1 -- 172.21.15.32:0/2050395180 wait complete. 2024-03-16T20:05:17.138 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:17.139 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:18.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 Processor -- start 2024-03-16T20:05:18.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 -- start start 2024-03-16T20:05:18.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e440c6420 0x7f2e440c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e441516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e44070760 con 0x7f2e440c6040 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e440731c0 con 0x7f2e440c6420 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.265+0000 7f2e4a6d8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440734a0 con 0x7f2e44158390 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e440c6420 0x7f2e440c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e48c75640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e441516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e440c6420 0x7f2e440c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60348/0 (socket says 172.21.15.32:60348) 2024-03-16T20:05:18.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 -- 172.21.15.32:0/2684898152 learned_addr learned my addr 172.21.15.32:0/2684898152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:18.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 -- 172.21.15.32:0/2684898152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 msgr2=0x7f2e441516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:18.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 --2- 172.21.15.32:0/2684898152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e441516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 -- 172.21.15.32:0/2684898152 >> v1:172.21.15.32:6789/0 conn(0x7f2e440c6040 legacy=0x7f2e44157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:18.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.269+0000 7f2e437fe640 1 -- 172.21.15.32:0/2684898152 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4406fe30 con 0x7f2e440c6420 2024-03-16T20:05:18.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.277+0000 7f2e437fe640 1 --2- 172.21.15.32:0/2684898152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6420 0x7f2e440c8800 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009870 tx=0x7f2e3801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73b273d3c31a003 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:18.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e42ffd640 1 -- 172.21.15.32:0/2684898152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e380aa610 con 0x7f2e440c6420 2024-03-16T20:05:18.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e42ffd640 1 -- 172.21.15.32:0/2684898152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e380a6040 con 0x7f2e440c6420 2024-03-16T20:05:18.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e42ffd640 1 -- 172.21.15.32:0/2684898152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e380aa990 con 0x7f2e440c6420 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/2684898152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6420 msgr2=0x7f2e440c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/2684898152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6420 0x7f2e440c8800 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009870 tx=0x7f2e3801c4f0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/2684898152 shutdown_connections 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/2684898152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e441516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/2684898152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6420 0x7f2e440c8800 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/2684898152 >> 172.21.15.32:0/2684898152 conn(0x7f2e4406c0d0 msgr2=0x7f2e4406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/2684898152 shutdown_connections 2024-03-16T20:05:18.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.281+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/2684898152 wait complete. 2024-03-16T20:05:18.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 Processor -- start 2024-03-16T20:05:18.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 -- start start 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e44143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e441f4c70 0x7f2e4413be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e441f1de0 con 0x7f2e440c6040 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4413e500 con 0x7f2e441f4c70 2024-03-16T20:05:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e4a6d8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4413e7e0 con 0x7f2e44158390 2024-03-16T20:05:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e43fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e44143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e43fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60360/0 (socket says 172.21.15.32:60360) 2024-03-16T20:05:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e43fff640 1 -- 172.21.15.32:0/1961881031 learned_addr learned my addr 172.21.15.32:0/1961881031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.285+0000 7f2e48c75640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e441f4c70 0x7f2e4413be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.289+0000 7f2e43fff640 1 -- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 msgr2=0x7f2e44143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.289+0000 7f2e43fff640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e44143b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.289+0000 7f2e43fff640 1 -- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e441f4c70 msgr2=0x7f2e4413be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.289+0000 7f2e43fff640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e441f4c70 0x7f2e4413be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.289+0000 7f2e43fff640 1 -- 172.21.15.32:0/1961881031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380094d0 con 0x7f2e440c6040 2024-03-16T20:05:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.293+0000 7f2e43fff640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c450 tx=0x7f2e3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:18.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34012830 con 0x7f2e440c6040 2024-03-16T20:05:18.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e340129b0 con 0x7f2e440c6040 2024-03-16T20:05:18.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4413eac0 con 0x7f2e440c6040 2024-03-16T20:05:18.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34012d60 con 0x7f2e440c6040 2024-03-16T20:05:18.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e441f4470 con 0x7f2e440c6040 2024-03-16T20:05:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.297+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e44068000 con 0x7f2e440c6040 2024-03-16T20:05:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.329+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2e3401c460 con 0x7f2e440c6040 2024-03-16T20:05:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.329+0000 7f2e40ff9640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 0x7f2e30069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.329+0000 7f2e437fe640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 0x7f2e30069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.329+0000 7f2e437fe640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 0x7f2e30069350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e380ae000 tx=0x7f2e380a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.629+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2e34055e90 con 0x7f2e440c6040 2024-03-16T20:05:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.629+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2e34052130 con 0x7f2e440c6040 2024-03-16T20:05:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.845+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e440c7d40 con 0x7f2e440c6040 2024-03-16T20:05:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.865+0000 7f2e40ff9640 1 -- 172.21.15.32:0/1961881031 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f2e340557f0 con 0x7f2e440c6040 2024-03-16T20:05:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:18.849 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:18.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 msgr2=0x7f2e30069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 0x7f2e30069350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e380ae000 tx=0x7f2e380a7000 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 msgr2=0x7f2e441f18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c450 tx=0x7f2e3400c910 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 shutdown_connections 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2e30066e90 0x7f2e30069350 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e441f4c70 0x7f2e4413be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2e44158390 0x7f2e44143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 --2- 172.21.15.32:0/1961881031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e440c6040 0x7f2e441f18a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 >> 172.21.15.32:0/1961881031 conn(0x7f2e4406c0d0 msgr2=0x7f2e4406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 shutdown_connections 2024-03-16T20:05:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:18.869+0000 7f2e4a6d8640 1 -- 172.21.15.32:0/1961881031 wait complete. 2024-03-16T20:05:19.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:19.472 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:20.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:20.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 Processor -- start 2024-03-16T20:05:20.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 -- start start 2024-03-16T20:05:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0d4151ef0 0x7ff0d40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0d40c97b0 0x7ff0d40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d4158c80 con 0x7ff0d40c8720 2024-03-16T20:05:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d40b0ad0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0dc7fd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d4157cb0 con 0x7ff0d40c97b0 2024-03-16T20:05:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0da599640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0d4151ef0 0x7ff0d40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0d9d98640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0d40c97b0 0x7ff0d40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0da599640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0d4151ef0 0x7ff0d40c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51468/0 (socket says 172.21.15.32:51468) 2024-03-16T20:05:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.609+0000 7ff0da599640 1 -- 172.21.15.32:0/3148328023 learned_addr learned my addr 172.21.15.32:0/3148328023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.613+0000 7ff0da599640 1 -- 172.21.15.32:0/3148328023 >> v2:172.21.15.59:3300/0 conn(0x7ff0d40c97b0 msgr2=0x7ff0d40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.613+0000 7ff0da599640 1 --2- 172.21.15.32:0/3148328023 >> v2:172.21.15.59:3300/0 conn(0x7ff0d40c97b0 0x7ff0d40c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.613+0000 7ff0da599640 1 -- 172.21.15.32:0/3148328023 >> v1:172.21.15.32:6789/0 conn(0x7ff0d40c8720 legacy=0x7ff0d415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.613+0000 7ff0da599640 1 -- 172.21.15.32:0/3148328023 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d415bfc0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.625+0000 7ff0da599640 1 --2- 172.21.15.32:0/3148328023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40c9270 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40cad10 tx=0x7ff0d001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f802388ed5e0be server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:20.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0d9597640 1 -- 172.21.15.32:0/3148328023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00a8a10 con 0x7ff0d4151ef0 2024-03-16T20:05:20.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0d9597640 1 -- 172.21.15.32:0/3148328023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d00a8b70 con 0x7ff0d4151ef0 2024-03-16T20:05:20.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0d9597640 1 -- 172.21.15.32:0/3148328023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00b0440 con 0x7ff0d4151ef0 2024-03-16T20:05:20.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/3148328023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 msgr2=0x7ff0d40c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:20.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/3148328023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40c9270 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40cad10 tx=0x7ff0d001ed10 comp rx=0 tx=0).stop 2024-03-16T20:05:20.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/3148328023 shutdown_connections 2024-03-16T20:05:20.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/3148328023 >> v2:172.21.15.59:3300/0 conn(0x7ff0d40c97b0 0x7ff0d40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:20.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/3148328023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40c9270 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:20.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/3148328023 >> 172.21.15.32:0/3148328023 conn(0x7ff0d4070f90 msgr2=0x7ff0d40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:20.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/3148328023 shutdown_connections 2024-03-16T20:05:20.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.629+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/3148328023 wait complete. 2024-03-16T20:05:20.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 Processor -- start 2024-03-16T20:05:20.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 -- start start 2024-03-16T20:05:20.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0d40c8720 0x7ff0d41f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d40c97b0 0x7ff0d40c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f3f60 con 0x7ff0d4151ef0 2024-03-16T20:05:20.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40c5be0 con 0x7ff0d40c97b0 2024-03-16T20:05:20.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dc7fd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b30b0 con 0x7ff0d40c8720 2024-03-16T20:05:20.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dad9a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dad9a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51474/0 (socket says 172.21.15.32:51474) 2024-03-16T20:05:20.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0dad9a640 1 -- 172.21.15.32:0/2610248951 learned_addr learned my addr 172.21.15.32:0/2610248951 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:20.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0d9d98640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d40c97b0 0x7ff0d40c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.633+0000 7ff0da599640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0d40c8720 0x7ff0d41f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.637+0000 7ff0dad9a640 1 -- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0d40c8720 msgr2=0x7ff0d41f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:20.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.637+0000 7ff0dad9a640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0d40c8720 0x7ff0d41f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:20.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.637+0000 7ff0dad9a640 1 -- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d40c97b0 msgr2=0x7ff0d40c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:20.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.637+0000 7ff0dad9a640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d40c97b0 0x7ff0d40c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:20.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.637+0000 7ff0dad9a640 1 -- 172.21.15.32:0/2610248951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d00094d0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.641+0000 7ff0dad9a640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00c7a0 tx=0x7ff0cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:20.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.645+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc003b80 con 0x7ff0d4151ef0 2024-03-16T20:05:20.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.645+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc003ce0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.645+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00aa90 con 0x7ff0d4151ef0 2024-03-16T20:05:20.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.645+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d41f36c0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.649+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d41f3c80 con 0x7ff0d4151ef0 2024-03-16T20:05:20.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.649+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d4068000 con 0x7ff0d4151ef0 2024-03-16T20:05:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.673+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0cc00abf0 con 0x7ff0d4151ef0 2024-03-16T20:05:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.673+0000 7ff0caffd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 0x7ff0b8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.673+0000 7ff0da599640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 0x7ff0b8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.673+0000 7ff0da599640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 0x7ff0b8069380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00b2000 tx=0x7ff0d00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:20.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:20.677+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff0cc055e10 con 0x7ff0d4151ef0 2024-03-16T20:05:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.045+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff0cc0520b0 con 0x7ff0d4151ef0 2024-03-16T20:05:21.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.249+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0d40c5d20 con 0x7ff0d4151ef0 2024-03-16T20:05:21.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0caffd640 1 -- 172.21.15.32:0/2610248951 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff0cc055770 con 0x7ff0d4151ef0 2024-03-16T20:05:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:21.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:21.246 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 msgr2=0x7ff0b8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 0x7ff0b8069380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00b2000 tx=0x7ff0d00aa070 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 msgr2=0x7ff0d40b2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00c7a0 tx=0x7ff0cc00cc60 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 shutdown_connections 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0b8066ec0 0x7ff0b8069380 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0d4151ef0 0x7ff0d40b2780 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d40c97b0 0x7ff0d40c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 --2- 172.21.15.32:0/2610248951 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0d40c8720 0x7ff0d41f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 >> 172.21.15.32:0/2610248951 conn(0x7ff0d4070f90 msgr2=0x7ff0d4156b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:21.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 shutdown_connections 2024-03-16T20:05:21.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:21.265+0000 7ff0dc7fd640 1 -- 172.21.15.32:0/2610248951 wait complete. 2024-03-16T20:05:21.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:21.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:22.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:22.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 Processor -- start 2024-03-16T20:05:22.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 -- start start 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ca0151000 0x7f7ca00c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca015a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ca00c8980 0x7f7ca00c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca0072f10 con 0x7f7ca00c8980 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca00c5000 con 0x7f7ca0151000 2024-03-16T20:05:22.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.001+0000 7f7ca8623640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00c8ec0 con 0x7f7ca00c36b0 2024-03-16T20:05:22.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ca00c8980 0x7f7ca00c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:22.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca5bbe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:22.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ca00c8980 0x7f7ca00c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51490/0 (socket says 172.21.15.32:51490) 2024-03-16T20:05:22.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/3719577309 learned_addr learned my addr 172.21.15.32:0/3719577309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:22.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca63bf640 1 --2- 172.21.15.32:0/3719577309 >> v2:172.21.15.59:3300/0 conn(0x7f7ca0151000 0x7f7ca00c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/3719577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 msgr2=0x7f7ca015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/3719577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/3719577309 >> v2:172.21.15.59:3300/0 conn(0x7f7ca0151000 msgr2=0x7f7ca00c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/3719577309 >> v2:172.21.15.59:3300/0 conn(0x7f7ca0151000 0x7f7ca00c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.005+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/3719577309 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca0072950 con 0x7f7ca00c8980 2024-03-16T20:05:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.009+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/3719577309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca00c8980 0x7f7ca00c9160 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00725d0 tx=0x7f7c9801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74ef227c2bff551 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:22.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca53bd640 1 -- 172.21.15.32:0/3719577309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980a5870 con 0x7f7ca00c8980 2024-03-16T20:05:22.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca53bd640 1 -- 172.21.15.32:0/3719577309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c980a59d0 con 0x7f7ca00c8980 2024-03-16T20:05:22.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca53bd640 1 -- 172.21.15.32:0/3719577309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980a5d60 con 0x7f7ca00c8980 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- 172.21.15.32:0/3719577309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca00c8980 msgr2=0x7f7ca00c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 --2- 172.21.15.32:0/3719577309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca00c8980 0x7f7ca00c9160 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00725d0 tx=0x7f7c9801c9f0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- 172.21.15.32:0/3719577309 shutdown_connections 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 --2- 172.21.15.32:0/3719577309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca00c8980 0x7f7ca00c9160 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 --2- 172.21.15.32:0/3719577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 --2- 172.21.15.32:0/3719577309 >> v2:172.21.15.59:3300/0 conn(0x7f7ca0151000 0x7f7ca00c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- 172.21.15.32:0/3719577309 >> 172.21.15.32:0/3719577309 conn(0x7f7ca006c0d0 msgr2=0x7f7ca006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- 172.21.15.32:0/3719577309 shutdown_connections 2024-03-16T20:05:22.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- 172.21.15.32:0/3719577309 wait complete. 2024-03-16T20:05:23.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 Processor -- start 2024-03-16T20:05:23.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.017+0000 7f7ca8623640 1 -- start start 2024-03-16T20:05:23.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca01f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:23.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca00c8980 0x7f7ca014f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0148210 con 0x7f7ca0151000 2024-03-16T20:05:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0148ac0 con 0x7f7ca00c8980 2024-03-16T20:05:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca8623640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01f2f40 con 0x7f7ca00c36b0 2024-03-16T20:05:23.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca6bc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:23.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca5bbe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca00c8980 0x7f7ca014f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:23.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca63bf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca01f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:23.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca6bc0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51494/0 (socket says 172.21.15.32:51494) 2024-03-16T20:05:23.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.021+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/1566511365 learned_addr learned my addr 172.21.15.32:0/1566511365 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.025+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 msgr2=0x7f7ca01f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.025+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca01f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.025+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca00c8980 msgr2=0x7f7ca014f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.025+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca00c8980 0x7f7ca014f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.025+0000 7f7ca6bc0640 1 -- 172.21.15.32:0/1566511365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009b30 con 0x7f7ca0151000 2024-03-16T20:05:23.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.029+0000 7f7ca6bc0640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c980af040 tx=0x7f7c98019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980b8670 con 0x7f7ca0151000 2024-03-16T20:05:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c980b87d0 con 0x7f7ca0151000 2024-03-16T20:05:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980b8b80 con 0x7f7ca0151000 2024-03-16T20:05:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7ca8623640 1 -- 172.21.15.32:0/1566511365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca01f72c0 con 0x7f7ca0151000 2024-03-16T20:05:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7ca8623640 1 -- 172.21.15.32:0/1566511365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca01f3490 con 0x7f7ca0151000 2024-03-16T20:05:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.033+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c0052a0 con 0x7f7ca0151000 2024-03-16T20:05:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.061+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7c980b8ce0 con 0x7f7ca0151000 2024-03-16T20:05:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.061+0000 7f7c96ffd640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 0x7f7c84069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.061+0000 7f7ca63bf640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 0x7f7c84069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.065+0000 7f7ca63bf640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 0x7f7c84069300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b9e0 tx=0x7f7c9c007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:23.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.065+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7c980f15f0 con 0x7f7ca0151000 2024-03-16T20:05:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.321+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7c980ed890 con 0x7f7ca0151000 2024-03-16T20:05:23.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.577+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c7c005900 con 0x7f7ca0151000 2024-03-16T20:05:23.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.649+0000 7f7c96ffd640 1 -- 172.21.15.32:0/1566511365 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f7c7c005900 con 0x7f7ca0151000 2024-03-16T20:05:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:23.632 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:23.632 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 msgr2=0x7f7c84069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 0x7f7c84069300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b9e0 tx=0x7f7c9c007040 comp rx=0 tx=0).stop 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 msgr2=0x7f7ca0160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c980af040 tx=0x7f7c98019790 comp rx=0 tx=0).stop 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 shutdown_connections 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c84066e40 0x7f7c84069300 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ca0151000 0x7f7ca0160b00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca00c8980 0x7f7ca014f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 --2- 172.21.15.32:0/1566511365 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ca00c36b0 0x7f7ca01f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:23.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 >> 172.21.15.32:0/1566511365 conn(0x7f7ca006c0d0 msgr2=0x7f7ca00c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:23.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 shutdown_connections 2024-03-16T20:05:23.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:23.653+0000 7f7c94ff9640 1 -- 172.21.15.32:0/1566511365 wait complete. 2024-03-16T20:05:24.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:24.262 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:25.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:25.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 Processor -- start 2024-03-16T20:05:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 -- start start 2024-03-16T20:05:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f938c1540c0 0x7f938c15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f938c0c3570 0x7f938c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938c06c460 con 0x7f938c15ac10 2024-03-16T20:05:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938c0c3f40 con 0x7f938c1540c0 2024-03-16T20:05:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.389+0000 7f939302a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938c15a240 con 0x7f938c0c3570 2024-03-16T20:05:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f938c1540c0 0x7f938c15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f938c1540c0 0x7f938c15a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51508/0 (socket says 172.21.15.32:51508) 2024-03-16T20:05:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 -- 172.21.15.32:0/75616382 learned_addr learned my addr 172.21.15.32:0/75616382 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f93915c7640 1 --2- 172.21.15.32:0/75616382 >> v2:172.21.15.59:3300/0 conn(0x7f938c0c3570 0x7f938c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 -- 172.21.15.32:0/75616382 >> v2:172.21.15.59:3300/0 conn(0x7f938c0c3570 msgr2=0x7f938c0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 --2- 172.21.15.32:0/75616382 >> v2:172.21.15.59:3300/0 conn(0x7f938c0c3570 0x7f938c0c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 -- 172.21.15.32:0/75616382 >> v1:172.21.15.59:6789/0 conn(0x7f938c15ac10 legacy=0x7f938c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.393+0000 7f9390dc6640 1 -- 172.21.15.32:0/75616382 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c06cf20 con 0x7f938c1540c0 2024-03-16T20:05:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.397+0000 7f9390dc6640 1 --2- 172.21.15.32:0/75616382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c15a6d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f938c06c740 tx=0x7f938001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a674140103cd0338 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:25.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f938b7fe640 1 -- 172.21.15.32:0/75616382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93800aa610 con 0x7f938c1540c0 2024-03-16T20:05:25.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f938b7fe640 1 -- 172.21.15.32:0/75616382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93800a6040 con 0x7f938c1540c0 2024-03-16T20:05:25.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f938b7fe640 1 -- 172.21.15.32:0/75616382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93800aa990 con 0x7f938c1540c0 2024-03-16T20:05:25.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- 172.21.15.32:0/75616382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 msgr2=0x7f938c15a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 --2- 172.21.15.32:0/75616382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c15a6d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f938c06c740 tx=0x7f938001c4b0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- 172.21.15.32:0/75616382 shutdown_connections 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 --2- 172.21.15.32:0/75616382 >> v2:172.21.15.59:3300/0 conn(0x7f938c0c3570 0x7f938c0c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 --2- 172.21.15.32:0/75616382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c15a6d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- 172.21.15.32:0/75616382 >> 172.21.15.32:0/75616382 conn(0x7f938c070f90 msgr2=0x7f938c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- 172.21.15.32:0/75616382 shutdown_connections 2024-03-16T20:05:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- 172.21.15.32:0/75616382 wait complete. 2024-03-16T20:05:25.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 Processor -- start 2024-03-16T20:05:25.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.405+0000 7f939302a640 1 -- start start 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f938c0c3570 0x7f938c1469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f938c15ac10 0x7f938c0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1f5cd0 con 0x7f938c1540c0 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0c6a20 con 0x7f938c15ac10 2024-03-16T20:05:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f939302a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0c6d90 con 0x7f938c0c3570 2024-03-16T20:05:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f938bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f93915c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f938c15ac10 0x7f938c0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f938bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51514/0 (socket says 172.21.15.32:51514) 2024-03-16T20:05:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f938bfff640 1 -- 172.21.15.32:0/2471111114 learned_addr learned my addr 172.21.15.32:0/2471111114 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.409+0000 7f9390dc6640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f938c0c3570 0x7f938c1469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.413+0000 7f938bfff640 1 -- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f938c0c3570 msgr2=0x7f938c1469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:25.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.413+0000 7f938bfff640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f938c0c3570 0x7f938c1469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.413+0000 7f938bfff640 1 -- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f938c15ac10 msgr2=0x7f938c0cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:25.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.413+0000 7f938bfff640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f938c15ac10 0x7f938c0cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:25.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.413+0000 7f938bfff640 1 -- 172.21.15.32:0/2471111114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93800094d0 con 0x7f938c1540c0 2024-03-16T20:05:25.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.417+0000 7f938bfff640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f937800c480 tx=0x7f937800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:25.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378012830 con 0x7f938c1540c0 2024-03-16T20:05:25.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93780129b0 con 0x7f938c1540c0 2024-03-16T20:05:25.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c0c7070 con 0x7f938c1540c0 2024-03-16T20:05:25.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378012d60 con 0x7f938c1540c0 2024-03-16T20:05:25.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c0c7630 con 0x7f938c1540c0 2024-03-16T20:05:25.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.421+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c070390 con 0x7f938c1540c0 2024-03-16T20:05:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.445+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f937801c460 con 0x7f938c1540c0 2024-03-16T20:05:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.445+0000 7f93897fa640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 0x7f9370069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.445+0000 7f9390dc6640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 0x7f9370069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.449+0000 7f9390dc6640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 0x7f9370069350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93800ae000 tx=0x7f93800b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:25.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.749+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9378055f00 con 0x7f938c1540c0 2024-03-16T20:05:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:25.749+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f93780521a0 con 0x7f938c1540c0 2024-03-16T20:05:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.009+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f938c1f5e10 con 0x7f938c1540c0 2024-03-16T20:05:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.029+0000 7f93897fa640 1 -- 172.21.15.32:0/2471111114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f938c1f5e10 con 0x7f938c1540c0 2024-03-16T20:05:26.013 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:26.013 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 msgr2=0x7f9370069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 0x7f9370069350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93800ae000 tx=0x7f93800b5040 comp rx=0 tx=0).stop 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 msgr2=0x7f938c0ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f937800c480 tx=0x7f937800c940 comp rx=0 tx=0).stop 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 shutdown_connections 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9370066e90 0x7f9370069350 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f938c15ac10 0x7f938c0cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c1540c0 0x7f938c0ccc70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 --2- 172.21.15.32:0/2471111114 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f938c0c3570 0x7f938c1469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 >> 172.21.15.32:0/2471111114 conn(0x7f938c070f90 msgr2=0x7f938c158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 shutdown_connections 2024-03-16T20:05:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:26.033+0000 7f939302a640 1 -- 172.21.15.32:0/2471111114 wait complete. 2024-03-16T20:05:26.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:26.643 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:27.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:27.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 Processor -- start 2024-03-16T20:05:27.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 -- start start 2024-03-16T20:05:27.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f81141540c0 0x7f811415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f81140c3570 0x7f81140c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811406c460 con 0x7f811415ac10 2024-03-16T20:05:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81140c3f40 con 0x7f81141540c0 2024-03-16T20:05:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8119640640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811415a240 con 0x7f81140c3570 2024-03-16T20:05:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f8112ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f81141540c0 0x7f811415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.781+0000 7f81137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f81140c3570 0x7f81140c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f81127fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f811415ac10 0x7f81140c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59244/0 (socket says 172.21.15.32:59244) 2024-03-16T20:05:27.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f81127fc640 1 -- 172.21.15.32:0/360254856 learned_addr learned my addr 172.21.15.32:0/360254856 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:27.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f8112ffd640 1 -- 172.21.15.32:0/360254856 >> v2:172.21.15.59:3300/0 conn(0x7f81140c3570 msgr2=0x7f81140c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:27.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f8112ffd640 1 --2- 172.21.15.32:0/360254856 >> v2:172.21.15.59:3300/0 conn(0x7f81140c3570 0x7f81140c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:27.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f8112ffd640 1 -- 172.21.15.32:0/360254856 >> v1:172.21.15.32:6789/0 conn(0x7f811415ac10 legacy=0x7f81140c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:27.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.785+0000 7f8112ffd640 1 -- 172.21.15.32:0/360254856 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811406cf20 con 0x7f81141540c0 2024-03-16T20:05:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.793+0000 7f8112ffd640 1 --2- 172.21.15.32:0/360254856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f811415a6d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f811406c740 tx=0x7f810401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4d1bfb4d9e25af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.797+0000 7f8111ffb640 1 -- 172.21.15.32:0/360254856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040ab650 con 0x7f81141540c0 2024-03-16T20:05:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.797+0000 7f8111ffb640 1 -- 172.21.15.32:0/360254856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81040ab7b0 con 0x7f81141540c0 2024-03-16T20:05:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.797+0000 7f8111ffb640 1 -- 172.21.15.32:0/360254856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040abb40 con 0x7f81141540c0 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- 172.21.15.32:0/360254856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 msgr2=0x7f811415a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- 172.21.15.32:0/360254856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f811415a6d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f811406c740 tx=0x7f810401c450 comp rx=0 tx=0).stop 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- 172.21.15.32:0/360254856 shutdown_connections 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- 172.21.15.32:0/360254856 >> v2:172.21.15.59:3300/0 conn(0x7f81140c3570 0x7f81140c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- 172.21.15.32:0/360254856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f811415a6d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- 172.21.15.32:0/360254856 >> 172.21.15.32:0/360254856 conn(0x7f8114070f90 msgr2=0x7f81140713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- 172.21.15.32:0/360254856 shutdown_connections 2024-03-16T20:05:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- 172.21.15.32:0/360254856 wait complete. 2024-03-16T20:05:27.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 Processor -- start 2024-03-16T20:05:27.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- start start 2024-03-16T20:05:27.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81140c3570 0x7f81140c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f811415ac10 0x7f81141f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81141f5bc0 con 0x7f81141540c0 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140ca790 con 0x7f81140c3570 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.801+0000 7f8119640640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8114148d50 con 0x7f811415ac10 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.805+0000 7f81127fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.805+0000 7f81127fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51522/0 (socket says 172.21.15.32:51522) 2024-03-16T20:05:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.805+0000 7f81127fc640 1 -- 172.21.15.32:0/1540467569 learned_addr learned my addr 172.21.15.32:0/1540467569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:27.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.805+0000 7f8112ffd640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81140c3570 0x7f81140c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.805+0000 7f81137fe640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f811415ac10 0x7f81141f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.809+0000 7f81127fc640 1 -- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f811415ac10 msgr2=0x7f81141f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.809+0000 7f81127fc640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f811415ac10 0x7f81141f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.809+0000 7f81127fc640 1 -- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81140c3570 msgr2=0x7f81140c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.809+0000 7f81127fc640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81140c3570 0x7f81140c82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.809+0000 7f81127fc640 1 -- 172.21.15.32:0/1540467569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8104009420 con 0x7f81141540c0 2024-03-16T20:05:27.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.813+0000 7f81127fc640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f810800c500 tx=0x7f810800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108005cf0 con 0x7f81141540c0 2024-03-16T20:05:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108005e50 con 0x7f81141540c0 2024-03-16T20:05:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8114149030 con 0x7f81141540c0 2024-03-16T20:05:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080072f0 con 0x7f81141540c0 2024-03-16T20:05:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81141495f0 con 0x7f81141540c0 2024-03-16T20:05:27.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.817+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8114070390 con 0x7f81141540c0 2024-03-16T20:05:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.849+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8108007450 con 0x7f81141540c0 2024-03-16T20:05:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.849+0000 7f80fb7fe640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 0x7f8100069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.849+0000 7f8112ffd640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 0x7f8100069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:27.849+0000 7f8112ffd640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 0x7f8100069300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f81040af000 tx=0x7f81040a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:28.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.145+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8108054fe0 con 0x7f81141540c0 2024-03-16T20:05:28.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.145+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8108051280 con 0x7f81141540c0 2024-03-16T20:05:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.349+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81141f5d00 con 0x7f81141540c0 2024-03-16T20:05:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.365+0000 7f80fb7fe640 1 -- 172.21.15.32:0/1540467569 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f8108054940 con 0x7f81141540c0 2024-03-16T20:05:28.348 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:28.348 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:28.348 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 msgr2=0x7f8100069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 0x7f8100069300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f81040af000 tx=0x7f81040a7000 comp rx=0 tx=0).stop 2024-03-16T20:05:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 msgr2=0x7f81141f6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f810800c500 tx=0x7f810800c9c0 comp rx=0 tx=0).stop 2024-03-16T20:05:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 shutdown_connections 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8100066e40 0x7f8100069300 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f811415ac10 0x7f81141f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81141540c0 0x7f81141f6010 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 --2- 172.21.15.32:0/1540467569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81140c3570 0x7f81140c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 >> 172.21.15.32:0/1540467569 conn(0x7f8114070f90 msgr2=0x7f8114159650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 shutdown_connections 2024-03-16T20:05:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:28.369+0000 7f8119640640 1 -- 172.21.15.32:0/1540467569 wait complete. 2024-03-16T20:05:28.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:28.970 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:29.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:30.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 Processor -- start 2024-03-16T20:05:30.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 -- start start 2024-03-16T20:05:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8500c3570 0x7fa8500c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85006c460 con 0x7fa85015ac10 2024-03-16T20:05:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8500c3f60 con 0x7fa8500c3570 2024-03-16T20:05:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa8567f6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85015a240 con 0x7fa8501540c0 2024-03-16T20:05:30.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa84ffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa84ffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:58580/0 (socket says 172.21.15.32:58580) 2024-03-16T20:05:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa84ffff640 1 -- 172.21.15.32:0/3641004428 learned_addr learned my addr 172.21.15.32:0/3641004428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.101+0000 7fa854d93640 1 --2- 172.21.15.32:0/3641004428 >> v2:172.21.15.59:3300/0 conn(0x7fa8500c3570 0x7fa8500c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.105+0000 7fa84ffff640 1 -- 172.21.15.32:0/3641004428 >> v2:172.21.15.59:3300/0 conn(0x7fa8500c3570 msgr2=0x7fa8500c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.105+0000 7fa84ffff640 1 --2- 172.21.15.32:0/3641004428 >> v2:172.21.15.59:3300/0 conn(0x7fa8500c3570 0x7fa8500c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.105+0000 7fa84ffff640 1 -- 172.21.15.32:0/3641004428 >> v1:172.21.15.59:6789/0 conn(0x7fa85015ac10 legacy=0x7fa8500c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.105+0000 7fa84ffff640 1 -- 172.21.15.32:0/3641004428 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85006cf20 con 0x7fa8501540c0 2024-03-16T20:05:30.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.109+0000 7fa84ffff640 1 --2- 172.21.15.32:0/3641004428 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa840009870 tx=0x7fa84001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfd3d318184404aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:30.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.117+0000 7fa84effd640 1 -- 172.21.15.32:0/3641004428 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400a95f0 con 0x7fa8501540c0 2024-03-16T20:05:30.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa84effd640 1 -- 172.21.15.32:0/3641004428 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400a9750 con 0x7fa8501540c0 2024-03-16T20:05:30.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa84effd640 1 -- 172.21.15.32:0/3641004428 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400a9ae0 con 0x7fa8501540c0 2024-03-16T20:05:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 -- 172.21.15.32:0/3641004428 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 msgr2=0x7fa85015a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 --2- 172.21.15.32:0/3641004428 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa840009870 tx=0x7fa84001c530 comp rx=0 tx=0).stop 2024-03-16T20:05:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 -- 172.21.15.32:0/3641004428 shutdown_connections 2024-03-16T20:05:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 --2- 172.21.15.32:0/3641004428 >> v2:172.21.15.59:3300/0 conn(0x7fa8500c3570 0x7fa8500c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 --2- 172.21.15.32:0/3641004428 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8501540c0 0x7fa85015a6d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 -- 172.21.15.32:0/3641004428 >> 172.21.15.32:0/3641004428 conn(0x7fa850070f90 msgr2=0x7fa8500713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:30.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 -- 172.21.15.32:0/3641004428 shutdown_connections 2024-03-16T20:05:30.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.121+0000 7fa8567f6640 1 -- 172.21.15.32:0/3641004428 wait complete. 2024-03-16T20:05:30.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 Processor -- start 2024-03-16T20:05:30.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 -- start start 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8500c3570 0x7fa850152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa85015ac10 0x7fa8501458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501f5f20 con 0x7fa8501540c0 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501f61c0 con 0x7fa8500c3570 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa8567f6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501f64a0 con 0x7fa85015ac10 2024-03-16T20:05:30.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa84f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa84f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51834/0 (socket says 172.21.15.32:51834) 2024-03-16T20:05:30.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa84f7fe640 1 -- 172.21.15.32:0/1858043223 learned_addr learned my addr 172.21.15.32:0/1858043223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:30.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa84ffff640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8500c3570 0x7fa850152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.125+0000 7fa854d93640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa85015ac10 0x7fa8501458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.129+0000 7fa84f7fe640 1 -- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa85015ac10 msgr2=0x7fa8501458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.129+0000 7fa84f7fe640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa85015ac10 0x7fa8501458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.129+0000 7fa84f7fe640 1 -- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8500c3570 msgr2=0x7fa850152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.129+0000 7fa84f7fe640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8500c3570 0x7fa850152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.133+0000 7fa84f7fe640 1 -- 172.21.15.32:0/1858043223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400094d0 con 0x7fa8501540c0 2024-03-16T20:05:30.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.133+0000 7fa84f7fe640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c430 tx=0x7fa84400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440052d0 con 0x7fa8501540c0 2024-03-16T20:05:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844005430 con 0x7fa8501540c0 2024-03-16T20:05:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844007260 con 0x7fa8501540c0 2024-03-16T20:05:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8501f6780 con 0x7fa8501540c0 2024-03-16T20:05:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8501f6d40 con 0x7fa8501540c0 2024-03-16T20:05:30.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.141+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850070390 con 0x7fa8501540c0 2024-03-16T20:05:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.177+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa8440073c0 con 0x7fa8501540c0 2024-03-16T20:05:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.177+0000 7fa84cff9640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 0x7fa83c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.181+0000 7fa84ffff640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 0x7fa83c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.181+0000 7fa84ffff640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 0x7fa83c069300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8400ae000 tx=0x7fa8400b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.469+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa844007670 con 0x7fa8501540c0 2024-03-16T20:05:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.469+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa8440511b0 con 0x7fa8501540c0 2024-03-16T20:05:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.741+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa850068000 con 0x7fa8501540c0 2024-03-16T20:05:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.777+0000 7fa84cff9640 1 -- 172.21.15.32:0/1858043223 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa844020090 con 0x7fa8501540c0 2024-03-16T20:05:30.758 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:30.758 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.777+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 msgr2=0x7fa83c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.777+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 0x7fa83c069300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8400ae000 tx=0x7fa8400b3040 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.777+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 msgr2=0x7fa850145370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.777+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c430 tx=0x7fa84400c8f0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 shutdown_connections 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa83c066e40 0x7fa83c069300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa85015ac10 0x7fa8501458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8501540c0 0x7fa850145370 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 --2- 172.21.15.32:0/1858043223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8500c3570 0x7fa850152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 >> 172.21.15.32:0/1858043223 conn(0x7fa850070f90 msgr2=0x7fa85015b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 shutdown_connections 2024-03-16T20:05:30.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:30.781+0000 7fa8567f6640 1 -- 172.21.15.32:0/1858043223 wait complete. 2024-03-16T20:05:31.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:31.382 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:32.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 Processor -- start 2024-03-16T20:05:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 -- start start 2024-03-16T20:05:32.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd80c0725e0 0x7fd80c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:32.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c151020 con 0x7fd80c15a420 2024-03-16T20:05:32.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c072da0 con 0x7fd80c150c40 2024-03-16T20:05:32.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd8119e3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80c15a000 con 0x7fd80c0725e0 2024-03-16T20:05:32.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd80b7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd80c15a420 0x7fd80c15a890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57608/0 (socket says 172.21.15.32:57608) 2024-03-16T20:05:32.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.513+0000 7fd80b7fe640 1 -- 172.21.15.32:0/826860699 learned_addr learned my addr 172.21.15.32:0/826860699 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:32.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.517+0000 7fd80a7fc640 1 --2- 172.21.15.32:0/826860699 >> v2:172.21.15.59:3300/0 conn(0x7fd80c0725e0 0x7fd80c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:32.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.517+0000 7fd809ffb640 1 -- 172.21.15.32:0/826860699 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233333109 0 0) 0x7fd80c151020 con 0x7fd80c15a420 2024-03-16T20:05:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.517+0000 7fd809ffb640 1 -- 172.21.15.32:0/826860699 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f4003530 con 0x7fd80c15a420 2024-03-16T20:05:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.521+0000 7fd80a7fc640 1 -- 172.21.15.32:0/826860699 >> v1:172.21.15.59:6789/0 conn(0x7fd80c150c40 legacy=0x7fd80c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.521+0000 7fd80a7fc640 1 -- 172.21.15.32:0/826860699 >> v1:172.21.15.32:6789/0 conn(0x7fd80c15a420 legacy=0x7fd80c15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.521+0000 7fd80a7fc640 1 -- 172.21.15.32:0/826860699 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c15e310 con 0x7fd80c0725e0 2024-03-16T20:05:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.525+0000 7fd80a7fc640 1 --2- 172.21.15.32:0/826860699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c073070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8000097c0 tx=0x7fd80001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a29348b85ccdee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd809ffb640 1 -- 172.21.15.32:0/826860699 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8000a6040 con 0x7fd80c0725e0 2024-03-16T20:05:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd809ffb640 1 -- 172.21.15.32:0/826860699 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8000aa600 con 0x7fd80c0725e0 2024-03-16T20:05:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd809ffb640 1 -- 172.21.15.32:0/826860699 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8000aa990 con 0x7fd80c0725e0 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 -- 172.21.15.32:0/826860699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 msgr2=0x7fd80c073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 --2- 172.21.15.32:0/826860699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c073070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8000097c0 tx=0x7fd80001c4a0 comp rx=0 tx=0).stop 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 -- 172.21.15.32:0/826860699 shutdown_connections 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 --2- 172.21.15.32:0/826860699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c073070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 -- 172.21.15.32:0/826860699 >> 172.21.15.32:0/826860699 conn(0x7fd80c06c0d0 msgr2=0x7fd80c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 -- 172.21.15.32:0/826860699 shutdown_connections 2024-03-16T20:05:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.533+0000 7fd8119e3640 1 -- 172.21.15.32:0/826860699 wait complete. 2024-03-16T20:05:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 Processor -- start 2024-03-16T20:05:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 -- start start 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80c150c40 0x7fd80c149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c1f6210 con 0x7fd80c15a420 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c1511e0 con 0x7fd80c0725e0 2024-03-16T20:05:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd8119e3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c148590 con 0x7fd80c150c40 2024-03-16T20:05:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd80b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd80b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51842/0 (socket says 172.21.15.32:51842) 2024-03-16T20:05:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd80b7fe640 1 -- 172.21.15.32:0/167197693 learned_addr learned my addr 172.21.15.32:0/167197693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd80a7fc640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80c150c40 0x7fd80c149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.537+0000 7fd80affd640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.541+0000 7fd80b7fe640 1 -- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80c150c40 msgr2=0x7fd80c149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.541+0000 7fd80b7fe640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80c150c40 0x7fd80c149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.541+0000 7fd80b7fe640 1 -- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 msgr2=0x7fd80c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.541+0000 7fd80b7fe640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c0cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.541+0000 7fd80b7fe640 1 -- 172.21.15.32:0/167197693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009420 con 0x7fd80c15a420 2024-03-16T20:05:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.545+0000 7fd80b7fe640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd80400cb40 tx=0x7fd80400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80400aca0 con 0x7fd80c15a420 2024-03-16T20:05:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80400ae00 con 0x7fd80c15a420 2024-03-16T20:05:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80c15d560 con 0x7fd80c15a420 2024-03-16T20:05:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804012620 con 0x7fd80c15a420 2024-03-16T20:05:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c148b50 con 0x7fd80c15a420 2024-03-16T20:05:32.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.549+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80c149350 con 0x7fd80c15a420 2024-03-16T20:05:32.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.589+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd804007070 con 0x7fd80c15a420 2024-03-16T20:05:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.589+0000 7fd7f3fff640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 0x7fd7f8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.589+0000 7fd80affd640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 0x7fd7f8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.589+0000 7fd80affd640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 0x7fd7f8069380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc001810 tx=0x7fd7fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.593+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd8040558a0 con 0x7fd80c15a420 2024-03-16T20:05:32.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:32.865+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd804051b40 con 0x7fd80c15a420 2024-03-16T20:05:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.085+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd80c149350 con 0x7fd80c15a420 2024-03-16T20:05:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.097+0000 7fd7f3fff640 1 -- 172.21.15.32:0/167197693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fd80c149350 con 0x7fd80c15a420 2024-03-16T20:05:33.080 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:33.080 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 msgr2=0x7fd7f8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 0x7fd7f8069380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc001810 tx=0x7fd7fc009040 comp rx=0 tx=0).stop 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 msgr2=0x7fd80c147130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd80400cb40 tx=0x7fd80400a060 comp rx=0 tx=0).stop 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 shutdown_connections 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd7f8066ec0 0x7fd7f8069380 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80c15a420 0x7fd80c147130 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd80c150c40 0x7fd80c149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:33.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 --2- 172.21.15.32:0/167197693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80c0725e0 0x7fd80c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:33.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 >> 172.21.15.32:0/167197693 conn(0x7fd80c06c0d0 msgr2=0x7fd80c14ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:33.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 shutdown_connections 2024-03-16T20:05:33.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:33.101+0000 7fd8119e3640 1 -- 172.21.15.32:0/167197693 wait complete. 2024-03-16T20:05:33.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:33.704 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:34.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.833+0000 7f1a977e2640 1 Processor -- start 2024-03-16T20:05:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.833+0000 7f1a977e2640 1 -- start start 2024-03-16T20:05:34.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a977e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1a900c6c40 0x7f1a9006e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a977e2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a90151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a977e2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a90157ef0 con 0x7f1a900c7020 2024-03-16T20:05:34.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a977e2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a90157c50 con 0x7f1a900c6c40 2024-03-16T20:05:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a977e2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9006eed0 con 0x7f1a90071a50 2024-03-16T20:05:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a94d7d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a900c7020 0x7f1a90158380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57614/0 (socket says 172.21.15.32:57614) 2024-03-16T20:05:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a94d7d640 1 -- 172.21.15.32:0/1937087208 learned_addr learned my addr 172.21.15.32:0/1937087208 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/1937087208 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a90151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.837+0000 7f1a9557e640 1 --2- 172.21.15.32:0/1937087208 >> v2:172.21.15.59:3300/0 conn(0x7f1a900c6c40 0x7f1a9006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a87fff640 1 -- 172.21.15.32:0/1937087208 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212531943 0 0) 0x7f1a90157ef0 con 0x7f1a900c7020 2024-03-16T20:05:34.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a87fff640 1 -- 172.21.15.32:0/1937087208 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c003530 con 0x7f1a900c7020 2024-03-16T20:05:34.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a95d7f640 1 -- 172.21.15.32:0/1937087208 >> v2:172.21.15.59:3300/0 conn(0x7f1a900c6c40 msgr2=0x7f1a9006e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:34.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/1937087208 >> v2:172.21.15.59:3300/0 conn(0x7f1a900c6c40 0x7f1a9006e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a95d7f640 1 -- 172.21.15.32:0/1937087208 >> v1:172.21.15.32:6789/0 conn(0x7f1a900c7020 legacy=0x7f1a90158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:34.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.841+0000 7f1a95d7f640 1 -- 172.21.15.32:0/1937087208 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a90072b50 con 0x7f1a90071a50 2024-03-16T20:05:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.845+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/1937087208 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a90151970 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009d90 tx=0x7f1a8801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da28fe2a09fd534 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:34.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a87fff640 1 -- 172.21.15.32:0/1937087208 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880a5880 con 0x7f1a90071a50 2024-03-16T20:05:34.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a87fff640 1 -- 172.21.15.32:0/1937087208 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a880a59e0 con 0x7f1a90071a50 2024-03-16T20:05:34.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a87fff640 1 -- 172.21.15.32:0/1937087208 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880a5d70 con 0x7f1a90071a50 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- 172.21.15.32:0/1937087208 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 msgr2=0x7f1a90151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 --2- 172.21.15.32:0/1937087208 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a90151970 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009d90 tx=0x7f1a8801c8b0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- 172.21.15.32:0/1937087208 shutdown_connections 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 --2- 172.21.15.32:0/1937087208 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a90151970 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 --2- 172.21.15.32:0/1937087208 >> v2:172.21.15.59:3300/0 conn(0x7f1a900c6c40 0x7f1a9006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- 172.21.15.32:0/1937087208 >> 172.21.15.32:0/1937087208 conn(0x7f1a900cbfe0 msgr2=0x7f1a900cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- 172.21.15.32:0/1937087208 shutdown_connections 2024-03-16T20:05:34.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- 172.21.15.32:0/1937087208 wait complete. 2024-03-16T20:05:34.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 Processor -- start 2024-03-16T20:05:34.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.853+0000 7f1a977e2640 1 -- start start 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a901f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a900c6c40 0x7f1a900c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90147ca0 con 0x7f1a900c7020 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a900c5500 con 0x7f1a900c6c40 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a977e2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90147eb0 con 0x7f1a90071a50 2024-03-16T20:05:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a95d7f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a95d7f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51850/0 (socket says 172.21.15.32:51850) 2024-03-16T20:05:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a95d7f640 1 -- 172.21.15.32:0/3407278124 learned_addr learned my addr 172.21.15.32:0/3407278124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a9557e640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a901f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.857+0000 7f1a94d7d640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a900c6c40 0x7f1a900c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.861+0000 7f1a95d7f640 1 -- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 msgr2=0x7f1a901f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:34.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.861+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a901f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.861+0000 7f1a95d7f640 1 -- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a900c6c40 msgr2=0x7f1a900c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:34.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.861+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a900c6c40 0x7f1a900c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:34.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.861+0000 7f1a95d7f640 1 -- 172.21.15.32:0/3407278124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a88009a70 con 0x7f1a900c7020 2024-03-16T20:05:34.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.865+0000 7f1a95d7f640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1a8801a4e0 tx=0x7f1a88019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880ad640 con 0x7f1a900c7020 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a880ad7a0 con 0x7f1a900c7020 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a90148190 con 0x7f1a900c7020 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880adb30 con 0x7f1a900c7020 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a901406e0 con 0x7f1a900c7020 2024-03-16T20:05:34.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.869+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a90068000 con 0x7f1a900c7020 2024-03-16T20:05:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.893+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1a880b4070 con 0x7f1a900c7020 2024-03-16T20:05:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.893+0000 7f1a85ffb640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 0x7f1a74069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.893+0000 7f1a9557e640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 0x7f1a74069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:34.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:34.893+0000 7f1a9557e640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 0x7f1a74069330 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009ac0 tx=0x7f1a8c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:35.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.189+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1a880f0780 con 0x7f1a900c7020 2024-03-16T20:05:35.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.189+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1a880eca20 con 0x7f1a900c7020 2024-03-16T20:05:35.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.513+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a90066f90 con 0x7f1a900c7020 2024-03-16T20:05:35.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a85ffb640 1 -- 172.21.15.32:0/3407278124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f1a880b2020 con 0x7f1a900c7020 2024-03-16T20:05:35.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:35.522 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 msgr2=0x7f1a74069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 0x7f1a74069330 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009ac0 tx=0x7f1a8c009190 comp rx=0 tx=0).stop 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 msgr2=0x7f1a900c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1a8801a4e0 tx=0x7f1a88019650 comp rx=0 tx=0).stop 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 shutdown_connections 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1a74066e70 0x7f1a74069330 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a900c7020 0x7f1a900c4fc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a900c6c40 0x7f1a900c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 --2- 172.21.15.32:0/3407278124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1a90071a50 0x7f1a901f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 >> 172.21.15.32:0/3407278124 conn(0x7f1a900cbfe0 msgr2=0x7f1a90156960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 shutdown_connections 2024-03-16T20:05:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:35.541+0000 7f1a977e2640 1 -- 172.21.15.32:0/3407278124 wait complete. 2024-03-16T20:05:36.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:36.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:37.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.277+0000 7f7b514ef640 1 Processor -- start 2024-03-16T20:05:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.277+0000 7f7b514ef640 1 -- start start 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7b4c151df0 0x7f7b4c072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b4c0c6320 0x7f7b4c158090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c073100 con 0x7f7b4c0c6320 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c070890 con 0x7f7b4c151df0 2024-03-16T20:05:37.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b514ef640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c073370 con 0x7f7b4c06ba60 2024-03-16T20:05:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b4b7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b4c0c6320 0x7f7b4c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b4b7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b4c0c6320 0x7f7b4c158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51854/0 (socket says 172.21.15.32:51854) 2024-03-16T20:05:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b4b7fe640 1 -- 172.21.15.32:0/4170591768 learned_addr learned my addr 172.21.15.32:0/4170591768 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b43fff640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.281+0000 7f7b4affd640 1 --2- 172.21.15.32:0/4170591768 >> v2:172.21.15.59:3300/0 conn(0x7f7b4c151df0 0x7f7b4c072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.285+0000 7f7b4b7fe640 1 -- 172.21.15.32:0/4170591768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 msgr2=0x7f7b4c0c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.285+0000 7f7b4b7fe640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.285+0000 7f7b4b7fe640 1 -- 172.21.15.32:0/4170591768 >> v2:172.21.15.59:3300/0 conn(0x7f7b4c151df0 msgr2=0x7f7b4c072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.285+0000 7f7b4b7fe640 1 --2- 172.21.15.32:0/4170591768 >> v2:172.21.15.59:3300/0 conn(0x7f7b4c151df0 0x7f7b4c072bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.285+0000 7f7b4b7fe640 1 -- 172.21.15.32:0/4170591768 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c06fff0 con 0x7f7b4c0c6320 2024-03-16T20:05:37.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.289+0000 7f7b4b7fe640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c158090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c06b630 tx=0x7f7b4401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151497362154aee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:37.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.293+0000 7f7b4a7fc640 1 -- 172.21.15.32:0/4170591768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b440af400 con 0x7f7b4c0c6320 2024-03-16T20:05:37.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.293+0000 7f7b4a7fc640 1 -- 172.21.15.32:0/4170591768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b440ab050 con 0x7f7b4c0c6320 2024-03-16T20:05:37.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.293+0000 7f7b4a7fc640 1 -- 172.21.15.32:0/4170591768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b440af780 con 0x7f7b4c0c6320 2024-03-16T20:05:37.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- 172.21.15.32:0/4170591768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 msgr2=0x7f7b4c158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:37.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c158090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c06b630 tx=0x7f7b4401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- 172.21.15.32:0/4170591768 shutdown_connections 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c158090 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- 172.21.15.32:0/4170591768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- 172.21.15.32:0/4170591768 >> v2:172.21.15.59:3300/0 conn(0x7f7b4c151df0 0x7f7b4c072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- 172.21.15.32:0/4170591768 >> 172.21.15.32:0/4170591768 conn(0x7f7b4c06c0d0 msgr2=0x7f7b4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- 172.21.15.32:0/4170591768 shutdown_connections 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- 172.21.15.32:0/4170591768 wait complete. 2024-03-16T20:05:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 Processor -- start 2024-03-16T20:05:37.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- start start 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c151df0 0x7f7b4c155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c153bc0 con 0x7f7b4c0c6320 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c153240 con 0x7f7b4c151df0 2024-03-16T20:05:37.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b514ef640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c152c80 con 0x7f7b4c06ba60 2024-03-16T20:05:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b43fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b43fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51858/0 (socket says 172.21.15.32:51858) 2024-03-16T20:05:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.297+0000 7f7b43fff640 1 -- 172.21.15.32:0/2774408694 learned_addr learned my addr 172.21.15.32:0/2774408694 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.301+0000 7f7b4affd640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.301+0000 7f7b4b7fe640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c151df0 0x7f7b4c155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.305+0000 7f7b43fff640 1 -- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 msgr2=0x7f7b4c0cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.305+0000 7f7b43fff640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.305+0000 7f7b43fff640 1 -- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c151df0 msgr2=0x7f7b4c155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.305+0000 7f7b43fff640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c151df0 0x7f7b4c155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.305+0000 7f7b43fff640 1 -- 172.21.15.32:0/2774408694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44009b30 con 0x7f7b4c0c6320 2024-03-16T20:05:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.309+0000 7f7b43fff640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e8b0 tx=0x7f7b3400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:37.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400c940 con 0x7f7b4c0c6320 2024-03-16T20:05:37.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3400caa0 con 0x7f7b4c0c6320 2024-03-16T20:05:37.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b514ef640 1 -- 172.21.15.32:0/2774408694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c152f60 con 0x7f7b4c0c6320 2024-03-16T20:05:37.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400ce30 con 0x7f7b4c0c6320 2024-03-16T20:05:37.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b514ef640 1 -- 172.21.15.32:0/2774408694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c152950 con 0x7f7b4c0c6320 2024-03-16T20:05:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.313+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b240052a0 con 0x7f7b4c0c6320 2024-03-16T20:05:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.337+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7b34013470 con 0x7f7b4c0c6320 2024-03-16T20:05:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.337+0000 7f7b437fe640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 0x7f7b2c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.337+0000 7f7b4affd640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 0x7f7b2c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.337+0000 7f7b4affd640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 0x7f7b2c069350 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009ac0 tx=0x7f7b3c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.609+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7b34055f40 con 0x7f7b4c0c6320 2024-03-16T20:05:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.609+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7b340521e0 con 0x7f7b4c0c6320 2024-03-16T20:05:37.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.857+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b24005900 con 0x7f7b4c0c6320 2024-03-16T20:05:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.937+0000 7f7b437fe640 1 -- 172.21.15.32:0/2774408694 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f7b24005900 con 0x7f7b4c0c6320 2024-03-16T20:05:37.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:37.920 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 msgr2=0x7f7b2c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 0x7f7b2c069350 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009ac0 tx=0x7f7b3c009190 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 msgr2=0x7f7b4c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e8b0 tx=0x7f7b3400ed70 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 shutdown_connections 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7b2c066e90 0x7f7b2c069350 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c151df0 0x7f7b4c155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c0c6320 0x7f7b4c154c00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 --2- 172.21.15.32:0/2774408694 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7b4c06ba60 0x7f7b4c0cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 >> 172.21.15.32:0/2774408694 conn(0x7f7b4c06c0d0 msgr2=0x7f7b4c14edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 shutdown_connections 2024-03-16T20:05:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:37.941+0000 7f7b417fa640 1 -- 172.21.15.32:0/2774408694 wait complete. 2024-03-16T20:05:38.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:38.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:39.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.685+0000 7fa5a3517640 1 Processor -- start 2024-03-16T20:05:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.685+0000 7fa5a3517640 1 -- start start 2024-03-16T20:05:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a3517640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 0x7fa59c153e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a3517640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59c15a390 con 0x7fa59c06de40 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a3517640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59c072bd0 con 0x7fa59c06e220 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a3517640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c071b70 con 0x7fa59c0c73f0 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a12b3640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa59c06de40 0x7fa59c15a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44568/0 (socket says 172.21.15.32:44568) 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a12b3640 1 -- 172.21.15.32:0/14234560 learned_addr learned my addr 172.21.15.32:0/14234560 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.689+0000 7fa5a0ab2640 1 --2- 172.21.15.32:0/14234560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 0x7fa59c153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.693+0000 7fa593fff640 1 -- 172.21.15.32:0/14234560 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760170886 0 0) 0x7fa59c15a390 con 0x7fa59c06de40 2024-03-16T20:05:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.693+0000 7fa593fff640 1 -- 172.21.15.32:0/14234560 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa588003530 con 0x7fa59c06de40 2024-03-16T20:05:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.693+0000 7fa5a0ab2640 1 -- 172.21.15.32:0/14234560 >> v1:172.21.15.59:6789/0 conn(0x7fa59c06e220 legacy=0x7fa59c164850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.693+0000 7fa5a0ab2640 1 -- 172.21.15.32:0/14234560 >> v1:172.21.15.32:6789/0 conn(0x7fa59c06de40 legacy=0x7fa59c15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.693+0000 7fa5a0ab2640 1 -- 172.21.15.32:0/14234560 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c072eb0 con 0x7fa59c0c73f0 2024-03-16T20:05:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.697+0000 7fa5a0ab2640 1 --2- 172.21.15.32:0/14234560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 0x7fa59c153e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0097f0 tx=0x7fa58c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=141a52b9c7f16e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:39.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.701+0000 7fa593fff640 1 -- 172.21.15.32:0/14234560 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0a89d0 con 0x7fa59c0c73f0 2024-03-16T20:05:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa593fff640 1 -- 172.21.15.32:0/14234560 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c0a8b30 con 0x7fa59c0c73f0 2024-03-16T20:05:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa593fff640 1 -- 172.21.15.32:0/14234560 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0a8ea0 con 0x7fa59c0c73f0 2024-03-16T20:05:39.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 -- 172.21.15.32:0/14234560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 msgr2=0x7fa59c153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 --2- 172.21.15.32:0/14234560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 0x7fa59c153e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0097f0 tx=0x7fa58c01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 -- 172.21.15.32:0/14234560 shutdown_connections 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 --2- 172.21.15.32:0/14234560 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c0c73f0 0x7fa59c153e50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 -- 172.21.15.32:0/14234560 >> 172.21.15.32:0/14234560 conn(0x7fa59c0cbfe0 msgr2=0x7fa59c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 -- 172.21.15.32:0/14234560 shutdown_connections 2024-03-16T20:05:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.705+0000 7fa5a3517640 1 -- 172.21.15.32:0/14234560 wait complete. 2024-03-16T20:05:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 Processor -- start 2024-03-16T20:05:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 -- start start 2024-03-16T20:05:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c06de40 0x7fa59c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa59c06e220 0x7fa59c1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c156a50 con 0x7fa59c0c1be0 2024-03-16T20:05:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0c17c0 con 0x7fa59c06e220 2024-03-16T20:05:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a3517640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0b0c30 con 0x7fa59c06de40 2024-03-16T20:05:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a1ab4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a12b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c06de40 0x7fa59c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a1ab4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52838/0 (socket says 172.21.15.32:52838) 2024-03-16T20:05:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a1ab4640 1 -- 172.21.15.32:0/1683399320 learned_addr learned my addr 172.21.15.32:0/1683399320 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.709+0000 7fa5a0ab2640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa59c06e220 0x7fa59c1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.713+0000 7fa5a1ab4640 1 -- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c06de40 msgr2=0x7fa59c155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.713+0000 7fa5a1ab4640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c06de40 0x7fa59c155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.713+0000 7fa5a1ab4640 1 -- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa59c06e220 msgr2=0x7fa59c1562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.713+0000 7fa5a1ab4640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa59c06e220 0x7fa59c1562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.713+0000 7fa5a1ab4640 1 -- 172.21.15.32:0/1683399320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c009420 con 0x7fa59c0c1be0 2024-03-16T20:05:39.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.717+0000 7fa5a1ab4640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa59400c730 tx=0x7fa59400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:39.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594003b80 con 0x7fa59c0c1be0 2024-03-16T20:05:39.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa594003ce0 con 0x7fa59c0c1be0 2024-03-16T20:05:39.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c0c25d0 con 0x7fa59c0c1be0 2024-03-16T20:05:39.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594005310 con 0x7fa59c0c1be0 2024-03-16T20:05:39.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c0c13f0 con 0x7fa59c0c1be0 2024-03-16T20:05:39.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.721+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c0c1900 con 0x7fa59c0c1be0 2024-03-16T20:05:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.753+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa594005680 con 0x7fa59c0c1be0 2024-03-16T20:05:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.753+0000 7fa591ffb640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 0x7fa580069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.753+0000 7fa5a12b3640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 0x7fa580069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:39.757+0000 7fa5a12b3640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 0x7fa580069350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa5980011f0 tx=0x7fa598007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.041+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa594055db0 con 0x7fa59c0c1be0 2024-03-16T20:05:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.041+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa594052050 con 0x7fa59c0c1be0 2024-03-16T20:05:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.281+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa59c0c1900 con 0x7fa59c0c1be0 2024-03-16T20:05:40.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.297+0000 7fa591ffb640 1 -- 172.21.15.32:0/1683399320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa59c0c1900 con 0x7fa59c0c1be0 2024-03-16T20:05:40.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:40.281 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:40.281 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 msgr2=0x7fa580069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 0x7fa580069350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa5980011f0 tx=0x7fa598007d40 comp rx=0 tx=0).stop 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 msgr2=0x7fa59c0c2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa59400c730 tx=0x7fa59400cbf0 comp rx=0 tx=0).stop 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 shutdown_connections 2024-03-16T20:05:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa580066e90 0x7fa580069350 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0c1be0 0x7fa59c0c2090 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa59c06e220 0x7fa59c1562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 --2- 172.21.15.32:0/1683399320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa59c06de40 0x7fa59c155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 >> 172.21.15.32:0/1683399320 conn(0x7fa59c0cbfe0 msgr2=0x7fa59c14df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 shutdown_connections 2024-03-16T20:05:40.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:40.301+0000 7fa5a3517640 1 -- 172.21.15.32:0/1683399320 wait complete. 2024-03-16T20:05:40.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:40.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:41.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 Processor -- start 2024-03-16T20:05:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 -- start start 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 0x7fc7700c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700ca1d0 con 0x7fc7700cae50 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700c34a0 con 0x7fc7700714f0 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.036+0000 7fc77836e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700c9ba0 con 0x7fc770071110 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.040+0000 7fc77690b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7700cae50 0x7fc770151a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44578/0 (socket says 172.21.15.32:44578) 2024-03-16T20:05:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.040+0000 7fc77690b640 1 -- 172.21.15.32:0/3381595945 learned_addr learned my addr 172.21.15.32:0/3381595945 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.040+0000 7fc77610a640 1 --2- 172.21.15.32:0/3381595945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 0x7fc7700c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.040+0000 7fc775108640 1 -- 172.21.15.32:0/3381595945 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43621141 0 0) 0x7fc7700ca1d0 con 0x7fc7700cae50 2024-03-16T20:05:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.040+0000 7fc775108640 1 -- 172.21.15.32:0/3381595945 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc75c003530 con 0x7fc7700cae50 2024-03-16T20:05:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.044+0000 7fc77610a640 1 -- 172.21.15.32:0/3381595945 >> v1:172.21.15.59:6789/0 conn(0x7fc7700714f0 legacy=0x7fc7700ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.044+0000 7fc77610a640 1 -- 172.21.15.32:0/3381595945 >> v1:172.21.15.32:6789/0 conn(0x7fc7700cae50 legacy=0x7fc770151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.044+0000 7fc77610a640 1 -- 172.21.15.32:0/3381595945 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700c2ee0 con 0x7fc770071110 2024-03-16T20:05:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.048+0000 7fc77610a640 1 --2- 172.21.15.32:0/3381595945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 0x7fc7700c82f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009820 tx=0x7fc76c01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd67a303487eee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc775108640 1 -- 172.21.15.32:0/3381595945 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0a89b0 con 0x7fc770071110 2024-03-16T20:05:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc775108640 1 -- 172.21.15.32:0/3381595945 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c0a8b10 con 0x7fc770071110 2024-03-16T20:05:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc775108640 1 -- 172.21.15.32:0/3381595945 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0a8ea0 con 0x7fc770071110 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 -- 172.21.15.32:0/3381595945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 msgr2=0x7fc7700c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 --2- 172.21.15.32:0/3381595945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 0x7fc7700c82f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009820 tx=0x7fc76c01ecb0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 -- 172.21.15.32:0/3381595945 shutdown_connections 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 --2- 172.21.15.32:0/3381595945 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc770071110 0x7fc7700c82f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 -- 172.21.15.32:0/3381595945 >> 172.21.15.32:0/3381595945 conn(0x7fc77006afc0 msgr2=0x7fc77006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 -- 172.21.15.32:0/3381595945 shutdown_connections 2024-03-16T20:05:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.056+0000 7fc77836e640 1 -- 172.21.15.32:0/3381595945 wait complete. 2024-03-16T20:05:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 Processor -- start 2024-03-16T20:05:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 -- start start 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7700714f0 0x7fc77014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77014d410 0x7fc77006ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77014dac0 con 0x7fc770071110 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77014bf80 con 0x7fc77014d410 2024-03-16T20:05:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77836e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7701e4930 con 0x7fc7700714f0 2024-03-16T20:05:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77610a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77690b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77014d410 0x7fc77006ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77610a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52852/0 (socket says 172.21.15.32:52852) 2024-03-16T20:05:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc77610a640 1 -- 172.21.15.32:0/522878600 learned_addr learned my addr 172.21.15.32:0/522878600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.060+0000 7fc775909640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7700714f0 0x7fc77014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.064+0000 7fc77610a640 1 -- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7700714f0 msgr2=0x7fc77014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.064+0000 7fc77610a640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7700714f0 0x7fc77014e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.064+0000 7fc77610a640 1 -- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77014d410 msgr2=0x7fc77006ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.064+0000 7fc77610a640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77014d410 0x7fc77006ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.064+0000 7fc77610a640 1 -- 172.21.15.32:0/522878600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c009420 con 0x7fc770071110 2024-03-16T20:05:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.068+0000 7fc77610a640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009820 tx=0x7fc76c01ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.072+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0b1d80 con 0x7fc770071110 2024-03-16T20:05:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.072+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c0b9410 con 0x7fc770071110 2024-03-16T20:05:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.072+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0b97c0 con 0x7fc770071110 2024-03-16T20:05:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.072+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7701e4c10 con 0x7fc770071110 2024-03-16T20:05:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.072+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77014c520 con 0x7fc770071110 2024-03-16T20:05:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.076+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7700c87b0 con 0x7fc770071110 2024-03-16T20:05:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.096+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc76c0b7040 con 0x7fc770071110 2024-03-16T20:05:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.096+0000 7fc766ffd640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 0x7fc754069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.096+0000 7fc775909640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 0x7fc754069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.096+0000 7fc775909640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 0x7fc754069350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc7700ce440 tx=0x7fc760009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:42.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.100+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc76c0f3ab0 con 0x7fc770071110 2024-03-16T20:05:42.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.364+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc76c0efd50 con 0x7fc770071110 2024-03-16T20:05:42.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:05:42.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:05:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.588+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc770068000 con 0x7fc770071110 2024-03-16T20:05:42.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc766ffd640 1 -- 172.21.15.32:0/522878600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc76c0b5070 con 0x7fc770071110 2024-03-16T20:05:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:42.598 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 msgr2=0x7fc754069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 0x7fc754069350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc7700ce440 tx=0x7fc760009040 comp rx=0 tx=0).stop 2024-03-16T20:05:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 msgr2=0x7fc77014e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009820 tx=0x7fc76c01ba50 comp rx=0 tx=0).stop 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 shutdown_connections 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc754066e90 0x7fc754069350 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77014d410 0x7fc77006ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7700714f0 0x7fc77014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 --2- 172.21.15.32:0/522878600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc770071110 0x7fc77014e000 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 >> 172.21.15.32:0/522878600 conn(0x7fc77006afc0 msgr2=0x7fc7700cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 shutdown_connections 2024-03-16T20:05:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:42.616+0000 7fc77836e640 1 -- 172.21.15.32:0/522878600 wait complete. 2024-03-16T20:05:43.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:43.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:44.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.348+0000 7f31b1528640 1 Processor -- start 2024-03-16T20:05:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.348+0000 7f31b1528640 1 -- start start 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31b1528640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31ac0c6420 0x7f31ac0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31b1528640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31ac158390 0x7f31ac1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31b1528640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac070760 con 0x7f31ac0c6040 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31b1528640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac0731c0 con 0x7f31ac158390 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31b1528640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac0734a0 con 0x7f31ac0c6420 2024-03-16T20:05:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31ab7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31ac158390 0x7f31ac1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31aa7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31ac0c6420 0x7f31ac0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31ab7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31ac158390 0x7f31ac1516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52860/0 (socket says 172.21.15.32:52860) 2024-03-16T20:05:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.352+0000 7f31ab7fe640 1 -- 172.21.15.32:0/507999323 learned_addr learned my addr 172.21.15.32:0/507999323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:44.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.356+0000 7f31ab7fe640 1 -- 172.21.15.32:0/507999323 >> v2:172.21.15.59:3300/0 conn(0x7f31ac0c6420 msgr2=0x7f31ac0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:44.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.356+0000 7f31ab7fe640 1 --2- 172.21.15.32:0/507999323 >> v2:172.21.15.59:3300/0 conn(0x7f31ac0c6420 0x7f31ac0c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:44.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.356+0000 7f31ab7fe640 1 -- 172.21.15.32:0/507999323 >> v1:172.21.15.59:6789/0 conn(0x7f31ac0c6040 legacy=0x7f31ac157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:44.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.356+0000 7f31ab7fe640 1 -- 172.21.15.32:0/507999323 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac06fe30 con 0x7f31ac158390 2024-03-16T20:05:44.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.380+0000 7f31ab7fe640 1 --2- 172.21.15.32:0/507999323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac158390 0x7f31ac1516d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f31ac06cf50 tx=0x7f31a401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9222e2a0195116 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:44.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31a9ffb640 1 -- 172.21.15.32:0/507999323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40a5840 con 0x7f31ac158390 2024-03-16T20:05:44.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31a9ffb640 1 -- 172.21.15.32:0/507999323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a40a59a0 con 0x7f31ac158390 2024-03-16T20:05:44.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31a9ffb640 1 -- 172.21.15.32:0/507999323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40a5d30 con 0x7f31ac158390 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- 172.21.15.32:0/507999323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac158390 msgr2=0x7f31ac1516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 --2- 172.21.15.32:0/507999323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac158390 0x7f31ac1516d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f31ac06cf50 tx=0x7f31a401c980 comp rx=0 tx=0).stop 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- 172.21.15.32:0/507999323 shutdown_connections 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 --2- 172.21.15.32:0/507999323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac158390 0x7f31ac1516d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 --2- 172.21.15.32:0/507999323 >> v2:172.21.15.59:3300/0 conn(0x7f31ac0c6420 0x7f31ac0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- 172.21.15.32:0/507999323 >> 172.21.15.32:0/507999323 conn(0x7f31ac06c0d0 msgr2=0x7f31ac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- 172.21.15.32:0/507999323 shutdown_connections 2024-03-16T20:05:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- 172.21.15.32:0/507999323 wait complete. 2024-03-16T20:05:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 Processor -- start 2024-03-16T20:05:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.388+0000 7f31b1528640 1 -- start start 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31ac0c6040 0x7f31ac142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31ac158390 0x7f31ac141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac141580 con 0x7f31ac0c6420 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac140080 con 0x7f31ac158390 2024-03-16T20:05:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31b1528640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac14a930 con 0x7f31ac0c6040 2024-03-16T20:05:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31aa7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31aa7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52862/0 (socket says 172.21.15.32:52862) 2024-03-16T20:05:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31aa7fc640 1 -- 172.21.15.32:0/4001745128 learned_addr learned my addr 172.21.15.32:0/4001745128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:44.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31ab7fe640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31ac158390 0x7f31ac141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.392+0000 7f31aaffd640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31ac0c6040 0x7f31ac142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.396+0000 7f31aa7fc640 1 -- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31ac0c6040 msgr2=0x7f31ac142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.396+0000 7f31aa7fc640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31ac0c6040 0x7f31ac142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.396+0000 7f31aa7fc640 1 -- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31ac158390 msgr2=0x7f31ac141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.396+0000 7f31aa7fc640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31ac158390 0x7f31ac141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.396+0000 7f31aa7fc640 1 -- 172.21.15.32:0/4001745128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4009b30 con 0x7f31ac0c6420 2024-03-16T20:05:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.400+0000 7f31aa7fc640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f31a000e900 tx=0x7f31a000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.404+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0009690 con 0x7f31ac0c6420 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.404+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a000ca00 con 0x7f31ac0c6420 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.404+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac140320 con 0x7f31ac0c6420 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.404+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a000cdb0 con 0x7f31ac0c6420 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.404+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac14ac10 con 0x7f31ac0c6420 2024-03-16T20:05:44.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.408+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac068000 con 0x7f31ac0c6420 2024-03-16T20:05:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.440+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f31a0013470 con 0x7f31ac0c6420 2024-03-16T20:05:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.444+0000 7f31937fe640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 0x7f3198069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.444+0000 7f31aaffd640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 0x7f3198069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.444+0000 7f31aaffd640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 0x7f3198069350 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f319c007fd0 tx=0x7f319c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.740+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f31a0055f90 con 0x7f31ac0c6420 2024-03-16T20:05:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.740+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f31a0052230 con 0x7f31ac0c6420 2024-03-16T20:05:44.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:44.964+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31ac0c7d40 con 0x7f31ac0c6420 2024-03-16T20:05:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.052+0000 7f31937fe640 1 -- 172.21.15.32:0/4001745128 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f31a0021090 con 0x7f31ac0c6420 2024-03-16T20:05:45.037 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:45.037 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 msgr2=0x7f3198069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 0x7f3198069350 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f319c007fd0 tx=0x7f319c007e40 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 msgr2=0x7f31ac140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f31a000e900 tx=0x7f31a000edc0 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 shutdown_connections 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3198066e90 0x7f3198069350 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31ac158390 0x7f31ac141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31ac0c6420 0x7f31ac140b00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 --2- 172.21.15.32:0/4001745128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31ac0c6040 0x7f31ac142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 >> 172.21.15.32:0/4001745128 conn(0x7f31ac06c0d0 msgr2=0x7f31ac06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 shutdown_connections 2024-03-16T20:05:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:45.056+0000 7f31b1528640 1 -- 172.21.15.32:0/4001745128 wait complete. 2024-03-16T20:05:45.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:45.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:46.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:46.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.788+0000 7f65a1782640 1 Processor -- start 2024-03-16T20:05:46.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.788+0000 7f65a1782640 1 -- start start 2024-03-16T20:05:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f65a1782640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f659c15ab30 0x7f659c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f65a1782640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c072dd0 con 0x7f659c0725e0 2024-03-16T20:05:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f65a1782640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c15d790 con 0x7f659c150d70 2024-03-16T20:05:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f65a1782640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c15da50 con 0x7f659c15ab30 2024-03-16T20:05:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f659a7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f659c0725e0 0x7f659c15a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44594/0 (socket says 172.21.15.32:44594) 2024-03-16T20:05:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f659a7fc640 1 -- 172.21.15.32:0/3301213564 learned_addr learned my addr 172.21.15.32:0/3301213564 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.792+0000 7f659b7fe640 1 --2- 172.21.15.32:0/3301213564 >> v2:172.21.15.59:3300/0 conn(0x7f659c15ab30 0x7f659c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.796+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450841539 0 0) 0x7f659c072dd0 con 0x7f659c0725e0 2024-03-16T20:05:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.796+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6584003530 con 0x7f659c0725e0 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198258179 0 0) 0x7f6584003530 con 0x7f659c0725e0 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 >> v2:172.21.15.59:3300/0 conn(0x7f659c15ab30 msgr2=0x7f659c073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 --2- 172.21.15.32:0/3301213564 >> v2:172.21.15.59:3300/0 conn(0x7f659c15ab30 0x7f659c073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 >> v1:172.21.15.59:6789/0 conn(0x7f659c150d70 legacy=0x7f659c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c15dd30 con 0x7f659c0725e0 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f6590001680 con 0x7f659c0725e0 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 >> v1:172.21.15.32:6789/0 conn(0x7f659c0725e0 legacy=0x7f659c15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6584003980 0x7f6584003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 0x7f6584004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840052b0 0x7f658400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c15da50 con 0x7f65840052b0 2024-03-16T20:05:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658400c180 con 0x7f6584003980 2024-03-16T20:05:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658400c340 con 0x7f65840047d0 2024-03-16T20:05:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.800+0000 7f659a7fc640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 0x7f6584004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659b7fe640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840052b0 0x7f658400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659a7fc640 1 -- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6584003980 msgr2=0x7f6584003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:46.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659a7fc640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6584003980 0x7f6584003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659a7fc640 1 -- 172.21.15.32:0/3301213564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840052b0 msgr2=0x7f658400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659a7fc640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840052b0 0x7f658400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.804+0000 7f659a7fc640 1 -- 172.21.15.32:0/3301213564 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c15dd30 con 0x7f65840047d0 2024-03-16T20:05:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.808+0000 7f659b7fe640 1 -- 172.21.15.32:0/3301213564 reap_dead start 2024-03-16T20:05:46.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.812+0000 7f659a7fc640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 0x7f6584004c50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6590001060 tx=0x7f659001cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f44a55f2a780dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590005ae0 con 0x7f65840047d0 2024-03-16T20:05:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590005c60 con 0x7f65840047d0 2024-03-16T20:05:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f6599ffb640 1 -- 172.21.15.32:0/3301213564 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900a5890 con 0x7f65840047d0 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 msgr2=0x7f6584004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 0x7f6584004c50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6590001060 tx=0x7f659001cbb0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- 172.21.15.32:0/3301213564 shutdown_connections 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 --2- 172.21.15.32:0/3301213564 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65840047d0 0x7f6584004c50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- 172.21.15.32:0/3301213564 >> 172.21.15.32:0/3301213564 conn(0x7f659c06c0d0 msgr2=0x7f659c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- 172.21.15.32:0/3301213564 shutdown_connections 2024-03-16T20:05:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- 172.21.15.32:0/3301213564 wait complete. 2024-03-16T20:05:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 Processor -- start 2024-03-16T20:05:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- start start 2024-03-16T20:05:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c1525f0 0x7f659c1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c1f3740 0x7f659c148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1f6ff0 con 0x7f65840047d0 2024-03-16T20:05:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1f2de0 con 0x7f659c1525f0 2024-03-16T20:05:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.820+0000 7f65a1782640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1f2ad0 con 0x7f659c1f3740 2024-03-16T20:05:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.824+0000 7f659b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c1f3740 0x7f659c148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.824+0000 7f659affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.824+0000 7f659affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52872/0 (socket says 172.21.15.32:52872) 2024-03-16T20:05:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.824+0000 7f659affd640 1 -- 172.21.15.32:0/2897332108 learned_addr learned my addr 172.21.15.32:0/2897332108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.824+0000 7f659a7fc640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c1525f0 0x7f659c1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.828+0000 7f659affd640 1 -- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c1f3740 msgr2=0x7f659c148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.828+0000 7f659affd640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c1f3740 0x7f659c148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.828+0000 7f659affd640 1 -- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c1525f0 msgr2=0x7f659c1f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.828+0000 7f659affd640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c1525f0 0x7f659c1f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.828+0000 7f659affd640 1 -- 172.21.15.32:0/2897332108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c1540f0 con 0x7f65840047d0 2024-03-16T20:05:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.832+0000 7f659affd640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f658c00c4d0 tx=0x7f658c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c0052d0 con 0x7f65840047d0 2024-03-16T20:05:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c005450 con 0x7f65840047d0 2024-03-16T20:05:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c1f6a50 con 0x7f65840047d0 2024-03-16T20:05:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c012870 con 0x7f65840047d0 2024-03-16T20:05:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c1f6d10 con 0x7f65840047d0 2024-03-16T20:05:46.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.840+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c159960 con 0x7f65840047d0 2024-03-16T20:05:46.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.864+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f658c0129d0 con 0x7f65840047d0 2024-03-16T20:05:46.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.864+0000 7f65837fe640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 0x7f6588069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:46.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.864+0000 7f659a7fc640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 0x7f6588069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:46.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.868+0000 7f659a7fc640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 0x7f6588069350 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f65900ab000 tx=0x7f65900b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:46.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:46.868+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f658c055b80 con 0x7f65840047d0 2024-03-16T20:05:47.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.148+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f658c051e20 con 0x7f65840047d0 2024-03-16T20:05:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.464+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f659c068000 con 0x7f65840047d0 2024-03-16T20:05:47.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.488+0000 7f65837fe640 1 -- 172.21.15.32:0/2897332108 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f658c0554e0 con 0x7f65840047d0 2024-03-16T20:05:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:47.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 msgr2=0x7f6588069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 0x7f6588069350 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f65900ab000 tx=0x7f65900b1040 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 msgr2=0x7f659c14bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f658c00c4d0 tx=0x7f658c00c990 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 shutdown_connections 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6588066e90 0x7f6588069350 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c1f3740 0x7f659c148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c1525f0 0x7f659c1f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 --2- 172.21.15.32:0/2897332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840047d0 0x7f659c14bdf0 secure :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f658c00c4d0 tx=0x7f658c00c990 comp rx=0 tx=0).stop 2024-03-16T20:05:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 >> 172.21.15.32:0/2897332108 conn(0x7f659c06c0d0 msgr2=0x7f659c0c91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 shutdown_connections 2024-03-16T20:05:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:47.492+0000 7f65a1782640 1 -- 172.21.15.32:0/2897332108 wait complete. 2024-03-16T20:05:48.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:48.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:49.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:49.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.220+0000 7fe84c84e640 1 Processor -- start 2024-03-16T20:05:49.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.220+0000 7fe84c84e640 1 -- start start 2024-03-16T20:05:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84c84e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe8480ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84c84e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8480c2910 0x7fe8480cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84c84e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8480ca1d0 con 0x7fe848071110 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84c84e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8480c34a0 con 0x7fe8480c2910 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84c84e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8480c9ba0 con 0x7fe8480714f0 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe846d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8480c2910 0x7fe8480cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe845d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe8480ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84659c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe848071110 0x7fe8480c82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42980/0 (socket says 172.21.15.32:42980) 2024-03-16T20:05:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.224+0000 7fe84659c640 1 -- 172.21.15.32:0/4263532876 learned_addr learned my addr 172.21.15.32:0/4263532876 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.228+0000 7fe845d9b640 1 -- 172.21.15.32:0/4263532876 >> v2:172.21.15.32:3300/0 conn(0x7fe8480c2910 msgr2=0x7fe8480cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.228+0000 7fe845d9b640 1 --2- 172.21.15.32:0/4263532876 >> v2:172.21.15.32:3300/0 conn(0x7fe8480c2910 0x7fe8480cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.228+0000 7fe845d9b640 1 -- 172.21.15.32:0/4263532876 >> v1:172.21.15.32:6789/0 conn(0x7fe848071110 legacy=0x7fe8480c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.228+0000 7fe845d9b640 1 -- 172.21.15.32:0/4263532876 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8480c2ee0 con 0x7fe8480714f0 2024-03-16T20:05:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.232+0000 7fe845d9b640 1 --2- 172.21.15.32:0/4263532876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe8480ca740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0097c0 tx=0x7fe83c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f6c491bdeb8d4c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84559a640 1 -- 172.21.15.32:0/4263532876 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0a8990 con 0x7fe8480714f0 2024-03-16T20:05:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84559a640 1 -- 172.21.15.32:0/4263532876 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c0a8af0 con 0x7fe8480714f0 2024-03-16T20:05:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84559a640 1 -- 172.21.15.32:0/4263532876 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0a8e80 con 0x7fe8480714f0 2024-03-16T20:05:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 -- 172.21.15.32:0/4263532876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 msgr2=0x7fe8480ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 --2- 172.21.15.32:0/4263532876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe8480ca740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0097c0 tx=0x7fe83c01ec90 comp rx=0 tx=0).stop 2024-03-16T20:05:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 -- 172.21.15.32:0/4263532876 shutdown_connections 2024-03-16T20:05:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 --2- 172.21.15.32:0/4263532876 >> v2:172.21.15.32:3300/0 conn(0x7fe8480c2910 0x7fe8480cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 --2- 172.21.15.32:0/4263532876 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe8480ca740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 -- 172.21.15.32:0/4263532876 >> 172.21.15.32:0/4263532876 conn(0x7fe84806afc0 msgr2=0x7fe84806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 -- 172.21.15.32:0/4263532876 shutdown_connections 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.240+0000 7fe84c84e640 1 -- 172.21.15.32:0/4263532876 wait complete. 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 Processor -- start 2024-03-16T20:05:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 -- start start 2024-03-16T20:05:49.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe84806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8480c2910 0x7fe8481e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481e4d10 con 0x7fe848071110 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481f0110 con 0x7fe8480c2910 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84c84e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481f0620 con 0x7fe8480714f0 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52398/0 (socket says 172.21.15.32:52398) 2024-03-16T20:05:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe84659c640 1 -- 172.21.15.32:0/937753206 learned_addr learned my addr 172.21.15.32:0/937753206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe846d9d640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8480c2910 0x7fe8481e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.244+0000 7fe845d9b640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe84806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.248+0000 7fe84659c640 1 -- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 msgr2=0x7fe84806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.248+0000 7fe84659c640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe84806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.248+0000 7fe84659c640 1 -- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8480c2910 msgr2=0x7fe8481e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.248+0000 7fe84659c640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8480c2910 0x7fe8481e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.248+0000 7fe84659c640 1 -- 172.21.15.32:0/937753206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c009420 con 0x7fe848071110 2024-03-16T20:05:49.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.252+0000 7fe84659c640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c480 tx=0x7fe83800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:49.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.256+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838005cf0 con 0x7fe848071110 2024-03-16T20:05:49.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.256+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe838005e50 con 0x7fe848071110 2024-03-16T20:05:49.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.256+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8480b2e20 con 0x7fe848071110 2024-03-16T20:05:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.256+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838007290 con 0x7fe848071110 2024-03-16T20:05:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.256+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8481f0340 con 0x7fe848071110 2024-03-16T20:05:49.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.260+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe848068000 con 0x7fe848071110 2024-03-16T20:05:49.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.276+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe8380073f0 con 0x7fe848071110 2024-03-16T20:05:49.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.276+0000 7fe836ffd640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 0x7fe828069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:49.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.276+0000 7fe845d9b640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 0x7fe828069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:49.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.276+0000 7fe845d9b640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 0x7fe828069350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0b2000 tx=0x7fe83c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:49.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.536+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe838055020 con 0x7fe848071110 2024-03-16T20:05:49.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.536+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe8380512c0 con 0x7fe848071110 2024-03-16T20:05:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.784+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe848151ca0 con 0x7fe848071110 2024-03-16T20:05:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.888+0000 7fe836ffd640 1 -- 172.21.15.32:0/937753206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fe838054980 con 0x7fe848071110 2024-03-16T20:05:49.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:49.871 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:49.871 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 msgr2=0x7fe828069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 0x7fe828069350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0b2000 tx=0x7fe83c0aa070 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 msgr2=0x7fe84806ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c480 tx=0x7fe83800c940 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 shutdown_connections 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe828066e90 0x7fe828069350 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8480c2910 0x7fe8481e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8480714f0 0x7fe84806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 --2- 172.21.15.32:0/937753206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848071110 0x7fe84806ce80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 >> 172.21.15.32:0/937753206 conn(0x7fe84806afc0 msgr2=0x7fe84814c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 shutdown_connections 2024-03-16T20:05:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:49.892+0000 7fe84c84e640 1 -- 172.21.15.32:0/937753206 wait complete. 2024-03-16T20:05:50.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:50.500 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:51.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.624+0000 7f848d991640 1 Processor -- start 2024-03-16T20:05:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.624+0000 7f848d991640 1 -- start start 2024-03-16T20:05:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.624+0000 7f848d991640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f84880c5f40 0x7f848806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.624+0000 7f848d991640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84880c6320 0x7f84880c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f848d991640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8488073610 con 0x7f8488157bf0 2024-03-16T20:05:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f848d991640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84880c7d70 con 0x7f84880c6320 2024-03-16T20:05:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f848d991640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848806b500 con 0x7f84880c5f40 2024-03-16T20:05:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f84867fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84880c6320 0x7f84880c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f84877fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8488157bf0 0x7f84881517f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42990/0 (socket says 172.21.15.32:42990) 2024-03-16T20:05:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f84877fe640 1 -- 172.21.15.32:0/636405725 learned_addr learned my addr 172.21.15.32:0/636405725 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.628+0000 7f8486ffd640 1 --2- 172.21.15.32:0/636405725 >> v2:172.21.15.59:3300/0 conn(0x7f84880c5f40 0x7f848806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.632+0000 7f84867fc640 1 -- 172.21.15.32:0/636405725 >> v2:172.21.15.59:3300/0 conn(0x7f84880c5f40 msgr2=0x7f848806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:51.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.632+0000 7f84867fc640 1 --2- 172.21.15.32:0/636405725 >> v2:172.21.15.59:3300/0 conn(0x7f84880c5f40 0x7f848806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:51.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.632+0000 7f84867fc640 1 -- 172.21.15.32:0/636405725 >> v1:172.21.15.32:6789/0 conn(0x7f8488157bf0 legacy=0x7f84881517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:51.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.632+0000 7f84867fc640 1 -- 172.21.15.32:0/636405725 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84880c6fd0 con 0x7f84880c6320 2024-03-16T20:05:51.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.636+0000 7f84867fc640 1 --2- 172.21.15.32:0/636405725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c88e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f84880c71c0 tx=0x7f847c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c022294d050ee83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f8485ffb640 1 -- 172.21.15.32:0/636405725 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0a8950 con 0x7f84880c6320 2024-03-16T20:05:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f8485ffb640 1 -- 172.21.15.32:0/636405725 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c0a8ab0 con 0x7f84880c6320 2024-03-16T20:05:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f8485ffb640 1 -- 172.21.15.32:0/636405725 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0a8e40 con 0x7f84880c6320 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 -- 172.21.15.32:0/636405725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 msgr2=0x7f84880c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 --2- 172.21.15.32:0/636405725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c88e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f84880c71c0 tx=0x7f847c01ec50 comp rx=0 tx=0).stop 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 -- 172.21.15.32:0/636405725 shutdown_connections 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 --2- 172.21.15.32:0/636405725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c88e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 --2- 172.21.15.32:0/636405725 >> v2:172.21.15.59:3300/0 conn(0x7f84880c5f40 0x7f848806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 -- 172.21.15.32:0/636405725 >> 172.21.15.32:0/636405725 conn(0x7f84880cc7a0 msgr2=0x7f84880ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 -- 172.21.15.32:0/636405725 shutdown_connections 2024-03-16T20:05:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.640+0000 7f848d991640 1 -- 172.21.15.32:0/636405725 wait complete. 2024-03-16T20:05:51.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 Processor -- start 2024-03-16T20:05:51.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 -- start start 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84880c5f40 0x7f84880c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8488157bf0 0x7f84881f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881f0930 con 0x7f84880c6320 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488147b80 con 0x7f8488157bf0 2024-03-16T20:05:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f848d991640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84880c5aa0 con 0x7f84880c5f40 2024-03-16T20:05:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f84867fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f84867fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52408/0 (socket says 172.21.15.32:52408) 2024-03-16T20:05:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f84867fc640 1 -- 172.21.15.32:0/182409015 learned_addr learned my addr 172.21.15.32:0/182409015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f8486ffd640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84880c5f40 0x7f84880c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.644+0000 7f84877fe640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8488157bf0 0x7f84881f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 -- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84880c5f40 msgr2=0x7f84880c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84880c5f40 0x7f84880c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 -- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8488157bf0 msgr2=0x7f84881f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8488157bf0 0x7f84881f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 -- 172.21.15.32:0/182409015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c009420 con 0x7f84880c6320 2024-03-16T20:05:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.652+0000 7f84867fc640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f847c0b3040 tx=0x7f847c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0b0dd0 con 0x7f84880c6320 2024-03-16T20:05:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c0b9430 con 0x7f84880c6320 2024-03-16T20:05:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84881efcb0 con 0x7f84880c6320 2024-03-16T20:05:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0b97e0 con 0x7f84880c6320 2024-03-16T20:05:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84881f0270 con 0x7f84880c6320 2024-03-16T20:05:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.660+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84881f0a70 con 0x7f84880c6320 2024-03-16T20:05:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.940+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f847c0b7040 con 0x7f84880c6320 2024-03-16T20:05:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.940+0000 7f846ffff640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 0x7f8474069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.940+0000 7f8486ffd640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 0x7f8474069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:51.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.944+0000 7f8486ffd640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 0x7f8474069330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8478009b60 tx=0x7f8478009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:51.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.984+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f847c0f3da0 con 0x7f84880c6320 2024-03-16T20:05:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:51.988+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f847c0f0040 con 0x7f84880c6320 2024-03-16T20:05:52.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.244+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8488068000 con 0x7f84880c6320 2024-03-16T20:05:52.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.268+0000 7f846ffff640 1 -- 172.21.15.32:0/182409015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f847c0b5020 con 0x7f84880c6320 2024-03-16T20:05:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:52.252 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:52.252 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.268+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 msgr2=0x7f8474069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.268+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 0x7f8474069330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8478009b60 tx=0x7f8478009190 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.268+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 msgr2=0x7f84880c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.268+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f847c0b3040 tx=0x7f847c01b9f0 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 shutdown_connections 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8474066e70 0x7f8474069330 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8488157bf0 0x7f84881f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84880c6320 0x7f84880c5010 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 --2- 172.21.15.32:0/182409015 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f84880c5f40 0x7f84880c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 >> 172.21.15.32:0/182409015 conn(0x7f84880cc7a0 msgr2=0x7f848814fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 shutdown_connections 2024-03-16T20:05:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:52.272+0000 7f848d991640 1 -- 172.21.15.32:0/182409015 wait complete. 2024-03-16T20:05:52.874 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:52.874 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:53.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:53.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 Processor -- start 2024-03-16T20:05:53.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 -- start start 2024-03-16T20:05:53.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60540c81e0 0x7f60540c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:53.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605406c460 con 0x7f60541540c0 2024-03-16T20:05:53.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605415a240 con 0x7f60540c3570 2024-03-16T20:05:53.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.012+0000 7f605a8b6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605406cf20 con 0x7f60540c81e0 2024-03-16T20:05:53.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.016+0000 7f60537fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60540c81e0 0x7f60540c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:53.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.016+0000 7f6053fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60541540c0 0x7f605415a6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43000/0 (socket says 172.21.15.32:43000) 2024-03-16T20:05:53.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.016+0000 7f6053fff640 1 -- 172.21.15.32:0/4226606934 learned_addr learned my addr 172.21.15.32:0/4226606934 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:54.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.020+0000 7f60537fe640 1 -- 172.21.15.32:0/4226606934 >> v1:172.21.15.59:6789/0 conn(0x7f60540c3570 legacy=0x7f60541647e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.020+0000 7f60537fe640 1 -- 172.21.15.32:0/4226606934 >> v1:172.21.15.32:6789/0 conn(0x7f60541540c0 legacy=0x7f605415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.020+0000 7f60537fe640 1 -- 172.21.15.32:0/4226606934 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605406d200 con 0x7f60540c81e0 2024-03-16T20:05:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.024+0000 7f60537fe640 1 --2- 172.21.15.32:0/4226606934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f60540c8630 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f60480097c0 tx=0x7f604801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ed127ee7011d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f6052ffd640 1 -- 172.21.15.32:0/4226606934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480ab670 con 0x7f60540c81e0 2024-03-16T20:05:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f6052ffd640 1 -- 172.21.15.32:0/4226606934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480ab7d0 con 0x7f60540c81e0 2024-03-16T20:05:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f6052ffd640 1 -- 172.21.15.32:0/4226606934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480abb60 con 0x7f60540c81e0 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 -- 172.21.15.32:0/4226606934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 msgr2=0x7f60540c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 --2- 172.21.15.32:0/4226606934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f60540c8630 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f60480097c0 tx=0x7f604801c470 comp rx=0 tx=0).stop 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 -- 172.21.15.32:0/4226606934 shutdown_connections 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 --2- 172.21.15.32:0/4226606934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f60540c8630 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 -- 172.21.15.32:0/4226606934 >> 172.21.15.32:0/4226606934 conn(0x7f6054070f90 msgr2=0x7f60540713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 -- 172.21.15.32:0/4226606934 shutdown_connections 2024-03-16T20:05:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.028+0000 7f605a8b6640 1 -- 172.21.15.32:0/4226606934 wait complete. 2024-03-16T20:05:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 Processor -- start 2024-03-16T20:05:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 -- start start 2024-03-16T20:05:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60540c3570 0x7f60541487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60541540c0 0x7f605414b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605414f340 con 0x7f60540c81e0 2024-03-16T20:05:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605414d6a0 con 0x7f60540c3570 2024-03-16T20:05:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f605a8b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60541f37f0 con 0x7f60541540c0 2024-03-16T20:05:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f60537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f60537fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52420/0 (socket says 172.21.15.32:52420) 2024-03-16T20:05:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f60537fe640 1 -- 172.21.15.32:0/240494467 learned_addr learned my addr 172.21.15.32:0/240494467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f6058e53640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60541540c0 0x7f605414b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.032+0000 7f6053fff640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60540c3570 0x7f60541487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.036+0000 7f60537fe640 1 -- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60541540c0 msgr2=0x7f605414b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.036+0000 7f60537fe640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60541540c0 0x7f605414b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.036+0000 7f60537fe640 1 -- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60540c3570 msgr2=0x7f60541487a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.036+0000 7f60537fe640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60540c3570 0x7f60541487a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.036+0000 7f60537fe640 1 -- 172.21.15.32:0/240494467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6048009420 con 0x7f60540c81e0 2024-03-16T20:05:54.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.040+0000 7f60537fe640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f60480ad040 tx=0x7f604801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:54.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.048+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b5810 con 0x7f60540c81e0 2024-03-16T20:05:54.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.048+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480b5990 con 0x7f60540c81e0 2024-03-16T20:05:54.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.048+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b5d40 con 0x7f60540c81e0 2024-03-16T20:05:54.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.048+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60541f3a90 con 0x7f60540c81e0 2024-03-16T20:05:54.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.048+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60540ccb00 con 0x7f60540c81e0 2024-03-16T20:05:54.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.052+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6054070390 con 0x7f60540c81e0 2024-03-16T20:05:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.404+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60480bf410 con 0x7f60540c81e0 2024-03-16T20:05:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.404+0000 7f6050ff9640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 0x7f6040069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.404+0000 7f6053fff640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 0x7f6040069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.404+0000 7f6053fff640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 0x7f6040069350 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6044000f70 tx=0x7f6044007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:54.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.416+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f60480f0ad0 con 0x7f60540c81e0 2024-03-16T20:05:54.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.416+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f60480ecd70 con 0x7f60540c81e0 2024-03-16T20:05:54.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.624+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6054068000 con 0x7f60540c81e0 2024-03-16T20:05:54.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.644+0000 7f6050ff9640 1 -- 172.21.15.32:0/240494467 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f60480f0430 con 0x7f60540c81e0 2024-03-16T20:05:54.629 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:54.629 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:54.629 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 msgr2=0x7f6040069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 0x7f6040069350 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6044000f70 tx=0x7f6044007e90 comp rx=0 tx=0).stop 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 msgr2=0x7f605414ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f60480ad040 tx=0x7f604801acb0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 shutdown_connections 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6040066e90 0x7f6040069350 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60541540c0 0x7f605414b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60540c81e0 0x7f605414ac80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 --2- 172.21.15.32:0/240494467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60540c3570 0x7f60541487a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 >> 172.21.15.32:0/240494467 conn(0x7f6054070f90 msgr2=0x7f605406f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 shutdown_connections 2024-03-16T20:05:54.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:54.648+0000 7f605a8b6640 1 -- 172.21.15.32:0/240494467 wait complete. 2024-03-16T20:05:55.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:55.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:56.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.376+0000 7f4378a44640 1 Processor -- start 2024-03-16T20:05:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.376+0000 7f4378a44640 1 -- start start 2024-03-16T20:05:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4378a44640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 0x7f43740c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4378a44640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4374156b30 0x7f43740c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4378a44640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740c7a40 con 0x7f43740c81b0 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4378a44640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43740c52e0 con 0x7f4374156b30 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4378a44640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740c9270 con 0x7f4374151000 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4372d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f43740c81b0 0x7f43740c3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43014/0 (socket says 172.21.15.32:43014) 2024-03-16T20:05:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4372d9d640 1 -- 172.21.15.32:0/2535627339 learned_addr learned my addr 172.21.15.32:0/2535627339 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f437259c640 1 --2- 172.21.15.32:0/2535627339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 0x7f43740c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.380+0000 7f4371d9b640 1 --2- 172.21.15.32:0/2535627339 >> v2:172.21.15.59:3300/0 conn(0x7f4374156b30 0x7f43740c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.384+0000 7f437159a640 1 -- 172.21.15.32:0/2535627339 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624358110 0 0) 0x7f43740c7a40 con 0x7f43740c81b0 2024-03-16T20:05:56.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.384+0000 7f437159a640 1 -- 172.21.15.32:0/2535627339 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c003530 con 0x7f43740c81b0 2024-03-16T20:05:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.388+0000 7f437259c640 1 -- 172.21.15.32:0/2535627339 >> v2:172.21.15.59:3300/0 conn(0x7f4374156b30 msgr2=0x7f43740c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.388+0000 7f437259c640 1 --2- 172.21.15.32:0/2535627339 >> v2:172.21.15.59:3300/0 conn(0x7f4374156b30 0x7f43740c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.388+0000 7f437259c640 1 -- 172.21.15.32:0/2535627339 >> v1:172.21.15.32:6789/0 conn(0x7f43740c81b0 legacy=0x7f43740c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.388+0000 7f437259c640 1 -- 172.21.15.32:0/2535627339 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740c9510 con 0x7f4374151000 2024-03-16T20:05:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.392+0000 7f437259c640 1 --2- 172.21.15.32:0/2535627339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 0x7f43740c70d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43640097c0 tx=0x7f436401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bd5b8ca49581cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.400+0000 7f437159a640 1 -- 172.21.15.32:0/2535627339 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640a8990 con 0x7f4374151000 2024-03-16T20:05:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.400+0000 7f437159a640 1 -- 172.21.15.32:0/2535627339 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43640a8af0 con 0x7f4374151000 2024-03-16T20:05:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.400+0000 7f437159a640 1 -- 172.21.15.32:0/2535627339 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640a8e80 con 0x7f4374151000 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- 172.21.15.32:0/2535627339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 msgr2=0x7f43740c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- 172.21.15.32:0/2535627339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 0x7f43740c70d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43640097c0 tx=0x7f436401ec90 comp rx=0 tx=0).stop 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- 172.21.15.32:0/2535627339 shutdown_connections 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- 172.21.15.32:0/2535627339 >> v2:172.21.15.59:3300/0 conn(0x7f4374156b30 0x7f43740c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- 172.21.15.32:0/2535627339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4374151000 0x7f43740c70d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- 172.21.15.32:0/2535627339 >> 172.21.15.32:0/2535627339 conn(0x7f437406c0d0 msgr2=0x7f437406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- 172.21.15.32:0/2535627339 shutdown_connections 2024-03-16T20:05:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- 172.21.15.32:0/2535627339 wait complete. 2024-03-16T20:05:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 Processor -- start 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- start start 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43740c81b0 0x7f437414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4374151000 0x7f43741404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43741ef700 con 0x7f4374156b30 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43741f2b40 con 0x7f4374151000 2024-03-16T20:05:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4378a44640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43741f2e00 con 0x7f43740c81b0 2024-03-16T20:05:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4372d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4371d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4374151000 0x7f43741404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4372d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52428/0 (socket says 172.21.15.32:52428) 2024-03-16T20:05:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.404+0000 7f4372d9d640 1 -- 172.21.15.32:0/2811478008 learned_addr learned my addr 172.21.15.32:0/2811478008 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:56.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.408+0000 7f437259c640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43740c81b0 0x7f437414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.412+0000 7f4372d9d640 1 -- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43740c81b0 msgr2=0x7f437414d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.412+0000 7f4372d9d640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43740c81b0 0x7f437414d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.412+0000 7f4372d9d640 1 -- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4374151000 msgr2=0x7f43741404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.412+0000 7f4372d9d640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4374151000 0x7f43741404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.412+0000 7f4372d9d640 1 -- 172.21.15.32:0/2811478008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364009420 con 0x7f4374156b30 2024-03-16T20:05:56.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.416+0000 7f4372d9d640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f436c00caa0 tx=0x7f436c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c00ac00 con 0x7f4374156b30 2024-03-16T20:05:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c00ad60 con 0x7f4374156b30 2024-03-16T20:05:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c012620 con 0x7f4374156b30 2024-03-16T20:05:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43741f30e0 con 0x7f4374156b30 2024-03-16T20:05:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43741f36a0 con 0x7f4374156b30 2024-03-16T20:05:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.420+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43741ef870 con 0x7f4374156b30 2024-03-16T20:05:56.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.460+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f436c007070 con 0x7f4374156b30 2024-03-16T20:05:56.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.460+0000 7f4362ffd640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 0x7f4354069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:56.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.460+0000 7f437259c640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 0x7f4354069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:56.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.460+0000 7f437259c640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 0x7f4354069330 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f43640b2000 tx=0x7f4364008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.744+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f436c055970 con 0x7f4374156b30 2024-03-16T20:05:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:56.744+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f436c051c10 con 0x7f4374156b30 2024-03-16T20:05:57.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.108+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4374068000 con 0x7f4374156b30 2024-03-16T20:05:57.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.124+0000 7f4362ffd640 1 -- 172.21.15.32:0/2811478008 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f436c017090 con 0x7f4374156b30 2024-03-16T20:05:57.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:57.109 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:57.110 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 msgr2=0x7f4354069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 0x7f4354069330 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f43640b2000 tx=0x7f4364008040 comp rx=0 tx=0).stop 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 msgr2=0x7f43741efb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f436c00caa0 tx=0x7f436c00cf60 comp rx=0 tx=0).stop 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 shutdown_connections 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4354066e70 0x7f4354069330 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4374156b30 0x7f43741efb90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4374151000 0x7f43741404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 --2- 172.21.15.32:0/2811478008 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43740c81b0 0x7f437414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:57.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 >> 172.21.15.32:0/2811478008 conn(0x7f437406c0d0 msgr2=0x7f4374157550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:57.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 shutdown_connections 2024-03-16T20:05:57.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:57.128+0000 7f4378a44640 1 -- 172.21.15.32:0/2811478008 wait complete. 2024-03-16T20:05:57.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:05:57.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:05:58.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:05:58.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 Processor -- start 2024-03-16T20:05:58.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 -- start start 2024-03-16T20:05:58.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b840c3c80 0x7f4b840c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8406c460 con 0x7f4b840c81e0 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8415a240 con 0x7f4b841540c0 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8c608640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8406cf20 con 0x7f4b840c3c80 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b8aba5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b840c3c80 0x7f4b840c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b89ba3640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4b840c81e0 0x7f4b840c3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43030/0 (socket says 172.21.15.32:43030) 2024-03-16T20:05:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.880+0000 7f4b89ba3640 1 -- 172.21.15.32:0/4025204526 learned_addr learned my addr 172.21.15.32:0/4025204526 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:05:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.884+0000 7f4b893a2640 1 -- 172.21.15.32:0/4025204526 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033041185 0 0) 0x7f4b8406c460 con 0x7f4b840c81e0 2024-03-16T20:05:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.884+0000 7f4b893a2640 1 -- 172.21.15.32:0/4025204526 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b70003530 con 0x7f4b840c81e0 2024-03-16T20:05:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.888+0000 7f4b8aba5640 1 -- 172.21.15.32:0/4025204526 >> v1:172.21.15.59:6789/0 conn(0x7f4b841540c0 legacy=0x7f4b8415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:05:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.888+0000 7f4b8aba5640 1 -- 172.21.15.32:0/4025204526 >> v1:172.21.15.32:6789/0 conn(0x7f4b840c81e0 legacy=0x7f4b840c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.888+0000 7f4b8aba5640 1 -- 172.21.15.32:0/4025204526 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8406d200 con 0x7f4b840c3c80 2024-03-16T20:05:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.892+0000 7f4b8aba5640 1 --2- 172.21.15.32:0/4025204526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c3c80 0x7f4b840c8630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009d90 tx=0x7f4b7c01c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=568d94eaf822ac4f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b893a2640 1 -- 172.21.15.32:0/4025204526 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0a59c0 con 0x7f4b840c3c80 2024-03-16T20:05:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b893a2640 1 -- 172.21.15.32:0/4025204526 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0a5b20 con 0x7f4b840c3c80 2024-03-16T20:05:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b893a2640 1 -- 172.21.15.32:0/4025204526 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0ad440 con 0x7f4b840c3c80 2024-03-16T20:05:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 -- 172.21.15.32:0/4025204526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c3c80 msgr2=0x7f4b840c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 --2- 172.21.15.32:0/4025204526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c3c80 0x7f4b840c8630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009d90 tx=0x7f4b7c01c900 comp rx=0 tx=0).stop 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 -- 172.21.15.32:0/4025204526 shutdown_connections 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 --2- 172.21.15.32:0/4025204526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c3c80 0x7f4b840c8630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 -- 172.21.15.32:0/4025204526 >> 172.21.15.32:0/4025204526 conn(0x7f4b84070f90 msgr2=0x7f4b840713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 -- 172.21.15.32:0/4025204526 shutdown_connections 2024-03-16T20:05:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.900+0000 7f4b8c608640 1 -- 172.21.15.32:0/4025204526 wait complete. 2024-03-16T20:05:58.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 Processor -- start 2024-03-16T20:05:58.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 -- start start 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4b840c3c80 0x7f4b8414d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c81e0 0x7f4b8414dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b840ccf70 con 0x7f4b841540c0 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84150a30 con 0x7f4b840c81e0 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8c608640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b841f5dc0 con 0x7f4b840c3c80 2024-03-16T20:05:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8aba5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b89ba3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c81e0 0x7f4b8414dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8aba5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52434/0 (socket says 172.21.15.32:52434) 2024-03-16T20:05:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8aba5640 1 -- 172.21.15.32:0/1603557639 learned_addr learned my addr 172.21.15.32:0/1603557639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:05:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.904+0000 7f4b8a3a4640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4b840c3c80 0x7f4b8414d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.912+0000 7f4b8aba5640 1 -- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4b840c3c80 msgr2=0x7f4b8414d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.912+0000 7f4b8aba5640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4b840c3c80 0x7f4b8414d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.912+0000 7f4b8aba5640 1 -- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c81e0 msgr2=0x7f4b8414dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.912+0000 7f4b8aba5640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c81e0 0x7f4b8414dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.912+0000 7f4b8aba5640 1 -- 172.21.15.32:0/1603557639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c009a70 con 0x7f4b841540c0 2024-03-16T20:05:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.916+0000 7f4b8aba5640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009d90 tx=0x7f4b7c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0adad0 con 0x7f4b841540c0 2024-03-16T20:05:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0adc30 con 0x7f4b841540c0 2024-03-16T20:05:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b841f60a0 con 0x7f4b841540c0 2024-03-16T20:05:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0b8420 con 0x7f4b841540c0 2024-03-16T20:05:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b841f6660 con 0x7f4b841540c0 2024-03-16T20:05:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.920+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b84070390 con 0x7f4b841540c0 2024-03-16T20:05:58.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.952+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4b7c0b8580 con 0x7f4b841540c0 2024-03-16T20:05:58.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.952+0000 7f4b7affd640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 0x7f4b68069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:05:58.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.952+0000 7f4b8a3a4640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 0x7f4b68069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:05:58.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:58.956+0000 7f4b8a3a4640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 0x7f4b68069330 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4b840c7df0 tx=0x7f4b80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:05:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.244+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4b7c0f1060 con 0x7f4b841540c0 2024-03-16T20:05:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.244+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4b7c0ed300 con 0x7f4b841540c0 2024-03-16T20:05:59.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.516+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b84068000 con 0x7f4b841540c0 2024-03-16T20:05:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.532+0000 7f4b7affd640 1 -- 172.21.15.32:0/1603557639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f4b7c0b2070 con 0x7f4b841540c0 2024-03-16T20:05:59.516 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:05:59.516 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:05:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:05:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 msgr2=0x7f4b68069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 0x7f4b68069330 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4b840c7df0 tx=0x7f4b80009040 comp rx=0 tx=0).stop 2024-03-16T20:05:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 msgr2=0x7f4b8414e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:05:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009d90 tx=0x7f4b7c0196a0 comp rx=0 tx=0).stop 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 shutdown_connections 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4b68066e70 0x7f4b68069330 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b841540c0 0x7f4b8414e250 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b840c81e0 0x7f4b8414dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 --2- 172.21.15.32:0/1603557639 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4b840c3c80 0x7f4b8414d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 >> 172.21.15.32:0/1603557639 conn(0x7f4b84070f90 msgr2=0x7f4b8406f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 shutdown_connections 2024-03-16T20:05:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:05:59.536+0000 7f4b8c608640 1 -- 172.21.15.32:0/1603557639 wait complete. 2024-03-16T20:06:00.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:00.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:01.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 Processor -- start 2024-03-16T20:06:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 -- start start 2024-03-16T20:06:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9d8154030 0x7fd9d8072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d80c9de0 0x7fd9d8159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d806b3f0 con 0x7fd9d8154410 2024-03-16T20:06:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d806b990 con 0x7fd9d8154030 2024-03-16T20:06:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.272+0000 7fd9dfcbd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d806b6b0 con 0x7fd9d80c9de0 2024-03-16T20:06:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.276+0000 7fd9dda59640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9d8154030 0x7fd9d8072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.276+0000 7fd9de25a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd9d8154410 0x7fd9d80cc620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42220/0 (socket says 172.21.15.32:42220) 2024-03-16T20:06:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.276+0000 7fd9de25a640 1 -- 172.21.15.32:0/941851130 learned_addr learned my addr 172.21.15.32:0/941851130 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.276+0000 7fd9dd258640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d80c9de0 0x7fd9d8159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.280+0000 7fd9dda59640 1 -- 172.21.15.32:0/941851130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d80c9de0 msgr2=0x7fd9d8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.280+0000 7fd9dda59640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d80c9de0 0x7fd9d8159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.280+0000 7fd9dda59640 1 -- 172.21.15.32:0/941851130 >> v1:172.21.15.32:6789/0 conn(0x7fd9d8154410 legacy=0x7fd9d80cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.280+0000 7fd9dda59640 1 -- 172.21.15.32:0/941851130 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80c9300 con 0x7fd9d8154030 2024-03-16T20:06:01.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.284+0000 7fd9dda59640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d8154030 0x7fd9d8072ec0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40097c0 tx=0x7fd9d401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41b131c4a9e9635 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.288+0000 7fd9dca57640 1 -- 172.21.15.32:0/941851130 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40a8990 con 0x7fd9d8154030 2024-03-16T20:06:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.288+0000 7fd9dca57640 1 -- 172.21.15.32:0/941851130 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d40a8af0 con 0x7fd9d8154030 2024-03-16T20:06:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.288+0000 7fd9dca57640 1 -- 172.21.15.32:0/941851130 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40a8e80 con 0x7fd9d8154030 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/941851130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d8154030 msgr2=0x7fd9d8072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d8154030 0x7fd9d8072ec0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40097c0 tx=0x7fd9d401ec90 comp rx=0 tx=0).stop 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/941851130 shutdown_connections 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d80c9de0 0x7fd9d8159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/941851130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d8154030 0x7fd9d8072ec0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/941851130 >> 172.21.15.32:0/941851130 conn(0x7fd9d806e6c0 msgr2=0x7fd9d806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/941851130 shutdown_connections 2024-03-16T20:06:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/941851130 wait complete. 2024-03-16T20:06:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 Processor -- start 2024-03-16T20:06:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- start start 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d8154030 0x7fd9d81f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9d8154410 0x7fd9d815d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d815cc70 con 0x7fd9d80c9de0 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d815c690 con 0x7fd9d8154410 2024-03-16T20:06:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dfcbd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d806cda0 con 0x7fd9d8154030 2024-03-16T20:06:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.292+0000 7fd9dda59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.296+0000 7fd9dda59640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55008/0 (socket says 172.21.15.32:55008) 2024-03-16T20:06:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.296+0000 7fd9dda59640 1 -- 172.21.15.32:0/2645157790 learned_addr learned my addr 172.21.15.32:0/2645157790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.296+0000 7fd9dd258640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d8154030 0x7fd9d81f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.296+0000 7fd9de25a640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9d8154410 0x7fd9d815d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.300+0000 7fd9dda59640 1 -- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d8154030 msgr2=0x7fd9d81f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.300+0000 7fd9dda59640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d8154030 0x7fd9d81f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.300+0000 7fd9dda59640 1 -- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9d8154410 msgr2=0x7fd9d815d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.300+0000 7fd9dda59640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9d8154410 0x7fd9d815d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.300+0000 7fd9dda59640 1 -- 172.21.15.32:0/2645157790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4009420 con 0x7fd9d80c9de0 2024-03-16T20:06:01.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.304+0000 7fd9dda59640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd9d401d340 tx=0x7fd9d401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40b1d40 con 0x7fd9d80c9de0 2024-03-16T20:06:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d40b9410 con 0x7fd9d80c9de0 2024-03-16T20:06:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40b1ea0 con 0x7fd9d80c9de0 2024-03-16T20:06:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d81f3b50 con 0x7fd9d80c9de0 2024-03-16T20:06:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d815a030 con 0x7fd9d80c9de0 2024-03-16T20:06:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.308+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d8068000 con 0x7fd9d80c9de0 2024-03-16T20:06:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.344+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd9d40b7040 con 0x7fd9d80c9de0 2024-03-16T20:06:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.344+0000 7fd9ce7fc640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 0x7fd9bc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:01.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.344+0000 7fd9dd258640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 0x7fd9bc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:01.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.344+0000 7fd9dd258640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 0x7fd9bc069380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd9d814a5c0 tx=0x7fd9c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.712+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd9d40f3970 con 0x7fd9d80c9de0 2024-03-16T20:06:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.712+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd9d40efc10 con 0x7fd9d80c9de0 2024-03-16T20:06:01.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.952+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d80c8bb0 con 0x7fd9d80c9de0 2024-03-16T20:06:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9ce7fc640 1 -- 172.21.15.32:0/2645157790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fd9d40f32d0 con 0x7fd9d80c9de0 2024-03-16T20:06:01.979 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:01.979 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 msgr2=0x7fd9bc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 0x7fd9bc069380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd9d814a5c0 tx=0x7fd9c8009190 comp rx=0 tx=0).stop 2024-03-16T20:06:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 msgr2=0x7fd9d81f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd9d401d340 tx=0x7fd9d401ba30 comp rx=0 tx=0).stop 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 shutdown_connections 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9bc066ec0 0x7fd9bc069380 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9d8154410 0x7fd9d815d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9d8154030 0x7fd9d81f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 --2- 172.21.15.32:0/2645157790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d80c9de0 0x7fd9d81f2f50 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 >> 172.21.15.32:0/2645157790 conn(0x7fd9d806e6c0 msgr2=0x7fd9d815a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 shutdown_connections 2024-03-16T20:06:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:01.996+0000 7fd9dfcbd640 1 -- 172.21.15.32:0/2645157790 wait complete. 2024-03-16T20:06:02.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:02.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:03.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.724+0000 7f36a7c93640 1 Processor -- start 2024-03-16T20:06:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.724+0000 7f36a7c93640 1 -- start start 2024-03-16T20:06:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a7c93640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36a006de40 0x7f36a015a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a7c93640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 0x7f36a0153e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a7c93640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a015a390 con 0x7f36a006e220 2024-03-16T20:06:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a7c93640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a0071b70 con 0x7f36a006de40 2024-03-16T20:06:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a7c93640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a0072bd0 con 0x7f36a00c73f0 2024-03-16T20:06:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a5a2f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36a006de40 0x7f36a015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a522e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 0x7f36a0153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a6230640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f36a006e220 0x7f36a015d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:48664/0 (socket says 172.21.15.32:48664) 2024-03-16T20:06:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.728+0000 7f36a6230640 1 -- 172.21.15.32:0/3478871642 learned_addr learned my addr 172.21.15.32:0/3478871642 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.736+0000 7f36a522e640 1 -- 172.21.15.32:0/3478871642 >> v2:172.21.15.59:3300/0 conn(0x7f36a006de40 msgr2=0x7f36a015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.736+0000 7f36a522e640 1 --2- 172.21.15.32:0/3478871642 >> v2:172.21.15.59:3300/0 conn(0x7f36a006de40 0x7f36a015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.736+0000 7f36a522e640 1 -- 172.21.15.32:0/3478871642 >> v1:172.21.15.59:6789/0 conn(0x7f36a006e220 legacy=0x7f36a015d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.736+0000 7f36a522e640 1 -- 172.21.15.32:0/3478871642 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0072eb0 con 0x7f36a00c73f0 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.740+0000 7f36a522e640 1 --2- 172.21.15.32:0/3478871642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 0x7f36a0153e50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36900097c0 tx=0x7f369001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831ebd07219f8b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a4a2d640 1 -- 172.21.15.32:0/3478871642 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900a8990 con 0x7f36a00c73f0 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a4a2d640 1 -- 172.21.15.32:0/3478871642 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36900a8af0 con 0x7f36a00c73f0 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a4a2d640 1 -- 172.21.15.32:0/3478871642 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900a8e80 con 0x7f36a00c73f0 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 -- 172.21.15.32:0/3478871642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 msgr2=0x7f36a0153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3478871642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 0x7f36a0153e50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36900097c0 tx=0x7f369001ec90 comp rx=0 tx=0).stop 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 -- 172.21.15.32:0/3478871642 shutdown_connections 2024-03-16T20:06:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3478871642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a00c73f0 0x7f36a0153e50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3478871642 >> v2:172.21.15.59:3300/0 conn(0x7f36a006de40 0x7f36a015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 -- 172.21.15.32:0/3478871642 >> 172.21.15.32:0/3478871642 conn(0x7f36a00cbfe0 msgr2=0x7f36a00cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 -- 172.21.15.32:0/3478871642 shutdown_connections 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.752+0000 7f36a7c93640 1 -- 172.21.15.32:0/3478871642 wait complete. 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 Processor -- start 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 -- start start 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a006de40 0x7f36a014c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36a006e220 0x7f36a014f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01ef800 con 0x7f36a00c3500 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01ef9f0 con 0x7f36a006e220 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a7c93640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a014e370 con 0x7f36a006de40 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a6230640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a5a2f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a006de40 0x7f36a014c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a6230640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55012/0 (socket says 172.21.15.32:55012) 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a6230640 1 -- 172.21.15.32:0/3472300121 learned_addr learned my addr 172.21.15.32:0/3472300121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.756+0000 7f36a522e640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36a006e220 0x7f36a014f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.764+0000 7f36a6230640 1 -- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a006de40 msgr2=0x7f36a014c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.764+0000 7f36a6230640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a006de40 0x7f36a014c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.764+0000 7f36a6230640 1 -- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36a006e220 msgr2=0x7f36a014f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.764+0000 7f36a6230640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36a006e220 0x7f36a014f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.764+0000 7f36a6230640 1 -- 172.21.15.32:0/3472300121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3690009420 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.768+0000 7f36a6230640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f369800caa0 tx=0x7f369800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369800ab40 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369800aca0 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698012620 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a01efc90 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a01f0250 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.772+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a00674e0 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.808+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3698007070 con 0x7f36a00c3500 2024-03-16T20:06:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.808+0000 7f36967fc640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 0x7f3684069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.808+0000 7f36a5a2f640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 0x7f3684069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.812+0000 7f36a5a2f640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 0x7f3684069300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f369c009b60 tx=0x7f369c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:03.812+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f36980558a0 con 0x7f36a00c3500 2024-03-16T20:06:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.112+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3698051b40 con 0x7f36a00c3500 2024-03-16T20:06:04.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.344+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36a0068000 con 0x7f36a00c3500 2024-03-16T20:06:04.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36967fc640 1 -- 172.21.15.32:0/3472300121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f3698017070 con 0x7f36a00c3500 2024-03-16T20:06:04.355 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:04.356 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:04.356 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:04.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 msgr2=0x7f3684069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:04.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 0x7f3684069300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f369c009b60 tx=0x7f369c009190 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 msgr2=0x7f36a014fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f369800caa0 tx=0x7f369800cf60 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 shutdown_connections 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3684066e40 0x7f3684069300 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a00c3500 0x7f36a014fee0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36a006e220 0x7f36a014f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 --2- 172.21.15.32:0/3472300121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36a006de40 0x7f36a014c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 >> 172.21.15.32:0/3472300121 conn(0x7f36a00cbfe0 msgr2=0x7f36a00cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 shutdown_connections 2024-03-16T20:06:04.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:04.372+0000 7f36a7c93640 1 -- 172.21.15.32:0/3472300121 wait complete. 2024-03-16T20:06:04.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:04.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:05.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:06.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 Processor -- start 2024-03-16T20:06:06.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 -- start start 2024-03-16T20:06:06.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d84151ef0 0x7f9d840c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d840c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d84158c80 con 0x7f9d840c8720 2024-03-16T20:06:06.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d840b0ad0 con 0x7f9d84151ef0 2024-03-16T20:06:06.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.120+0000 7f9d8c2cb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84157cb0 con 0x7f9d840c97b0 2024-03-16T20:06:06.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d89866640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d840c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d8a868640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9d840c8720 0x7f9d8415b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42224/0 (socket says 172.21.15.32:42224) 2024-03-16T20:06:06.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d8a868640 1 -- 172.21.15.32:0/1983291938 learned_addr learned my addr 172.21.15.32:0/1983291938 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:06.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d8a067640 1 --2- 172.21.15.32:0/1983291938 >> v2:172.21.15.59:3300/0 conn(0x7f9d84151ef0 0x7f9d840c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d89866640 1 -- 172.21.15.32:0/1983291938 >> v2:172.21.15.59:3300/0 conn(0x7f9d84151ef0 msgr2=0x7f9d840c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:06.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d89866640 1 --2- 172.21.15.32:0/1983291938 >> v2:172.21.15.59:3300/0 conn(0x7f9d84151ef0 0x7f9d840c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.124+0000 7f9d89866640 1 -- 172.21.15.32:0/1983291938 >> v1:172.21.15.32:6789/0 conn(0x7f9d840c8720 legacy=0x7f9d8415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.128+0000 7f9d89866640 1 -- 172.21.15.32:0/1983291938 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8415bfc0 con 0x7f9d840c97b0 2024-03-16T20:06:06.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.132+0000 7f9d89866640 1 --2- 172.21.15.32:0/1983291938 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d840c81e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d740097c0 tx=0x7f9d7401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abf31ab81af49ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.136+0000 7f9d89065640 1 -- 172.21.15.32:0/1983291938 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740a8990 con 0x7f9d840c97b0 2024-03-16T20:06:06.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.136+0000 7f9d89065640 1 -- 172.21.15.32:0/1983291938 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d740a8af0 con 0x7f9d840c97b0 2024-03-16T20:06:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/1983291938 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 msgr2=0x7f9d840c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/1983291938 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d840c81e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d740097c0 tx=0x7f9d7401ec90 comp rx=0 tx=0).stop 2024-03-16T20:06:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/1983291938 shutdown_connections 2024-03-16T20:06:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/1983291938 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d840c81e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/1983291938 >> v2:172.21.15.59:3300/0 conn(0x7f9d84151ef0 0x7f9d840c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/1983291938 >> 172.21.15.32:0/1983291938 conn(0x7f9d84070f90 msgr2=0x7f9d840713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/1983291938 shutdown_connections 2024-03-16T20:06:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/1983291938 wait complete. 2024-03-16T20:06:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 Processor -- start 2024-03-16T20:06:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- start start 2024-03-16T20:06:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d840c8720 0x7f9d841ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 0x7f9d8414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84152a40 con 0x7f9d84151ef0 2024-03-16T20:06:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8414cab0 con 0x7f9d840c8720 2024-03-16T20:06:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.140+0000 7f9d8c2cb640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841f0a80 con 0x7f9d840c97b0 2024-03-16T20:06:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.144+0000 7f9d89866640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.144+0000 7f9d8a868640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 0x7f9d8414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.144+0000 7f9d8a067640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d840c8720 0x7f9d841ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.144+0000 7f9d89866640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:35562/0 (socket says 172.21.15.32:35562) 2024-03-16T20:06:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.144+0000 7f9d89866640 1 -- 172.21.15.32:0/517172815 learned_addr learned my addr 172.21.15.32:0/517172815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d89866640 1 -- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d840c8720 msgr2=0x7f9d841ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d89866640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d840c8720 0x7f9d841ee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d89866640 1 -- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 msgr2=0x7f9d8414d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d89866640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 0x7f9d8414d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d89866640 1 -- 172.21.15.32:0/517172815 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d74009420 con 0x7f9d840c97b0 2024-03-16T20:06:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.148+0000 7f9d8a868640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 0x7f9d8414d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:06:06.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.152+0000 7f9d89866640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d740b3040 tx=0x7f9d7401ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740be070 con 0x7f9d840c97b0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74003620 con 0x7f9d840c97b0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8414cd00 con 0x7f9d840c97b0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740039b0 con 0x7f9d840c97b0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841f0d20 con 0x7f9d840c97b0 2024-03-16T20:06:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.160+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d84068000 con 0x7f9d840c97b0 2024-03-16T20:06:06.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.192+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9d740b7040 con 0x7f9d840c97b0 2024-03-16T20:06:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.192+0000 7f9d7affd640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 0x7f9d68069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.192+0000 7f9d8a067640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 0x7f9d68069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.192+0000 7f9d8a067640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 0x7f9d68069300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009ac0 tx=0x7f9d80009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:06.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.496+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9d740f3290 con 0x7f9d840c97b0 2024-03-16T20:06:06.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.536+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9d740ef530 con 0x7f9d840c97b0 2024-03-16T20:06:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.780+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d84070a40 con 0x7f9d840c97b0 2024-03-16T20:06:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d7affd640 1 -- 172.21.15.32:0/517172815 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f9d740b5070 con 0x7f9d840c97b0 2024-03-16T20:06:06.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:06.788 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:06.788 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 msgr2=0x7f9d68069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 0x7f9d68069300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009ac0 tx=0x7f9d80009190 comp rx=0 tx=0).stop 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 msgr2=0x7f9d841f1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d740b3040 tx=0x7f9d7401ba30 comp rx=0 tx=0).stop 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 shutdown_connections 2024-03-16T20:06:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9d68066e40 0x7f9d68069300 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d84151ef0 0x7f9d8414d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d840c97b0 0x7f9d841f1520 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 --2- 172.21.15.32:0/517172815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d840c8720 0x7f9d841ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 >> 172.21.15.32:0/517172815 conn(0x7f9d84070f90 msgr2=0x7f9d840c3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 shutdown_connections 2024-03-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:06.804+0000 7f9d8c2cb640 1 -- 172.21.15.32:0/517172815 wait complete. 2024-03-16T20:06:07.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:07.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:08.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 Processor -- start 2024-03-16T20:06:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 -- start start 2024-03-16T20:06:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8e6c150d70 0x7f8e6c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e6c0725e0 0x7f8e6c070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c072dd0 con 0x7f8e6c0708a0 2024-03-16T20:06:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c070cb0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e73c47640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c15d790 con 0x7f8e6c150d70 2024-03-16T20:06:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e711e2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e6c0725e0 0x7f8e6c070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e711e2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e6c0725e0 0x7f8e6c070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55024/0 (socket says 172.21.15.32:55024) 2024-03-16T20:06:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e711e2640 1 -- 172.21.15.32:0/4060739242 learned_addr learned my addr 172.21.15.32:0/4060739242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.544+0000 7f8e719e3640 1 --2- 172.21.15.32:0/4060739242 >> v2:172.21.15.59:3300/0 conn(0x7f8e6c150d70 0x7f8e6c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.548+0000 7f8e711e2640 1 -- 172.21.15.32:0/4060739242 >> v2:172.21.15.59:3300/0 conn(0x7f8e6c150d70 msgr2=0x7f8e6c073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.548+0000 7f8e711e2640 1 --2- 172.21.15.32:0/4060739242 >> v2:172.21.15.59:3300/0 conn(0x7f8e6c150d70 0x7f8e6c073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.548+0000 7f8e711e2640 1 -- 172.21.15.32:0/4060739242 >> v1:172.21.15.59:6789/0 conn(0x7f8e6c0708a0 legacy=0x7f8e6c15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.548+0000 7f8e711e2640 1 -- 172.21.15.32:0/4060739242 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c15da50 con 0x7f8e6c0725e0 2024-03-16T20:06:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.552+0000 7f8e711e2640 1 --2- 172.21.15.32:0/4060739242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c070360 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c06c080 tx=0x7f8e5c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed712d5d8cf44bc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.556+0000 7f8e709e1640 1 -- 172.21.15.32:0/4060739242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0ad640 con 0x7f8e6c0725e0 2024-03-16T20:06:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.556+0000 7f8e709e1640 1 -- 172.21.15.32:0/4060739242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c0ad7a0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.556+0000 7f8e709e1640 1 -- 172.21.15.32:0/4060739242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0adb30 con 0x7f8e6c0725e0 2024-03-16T20:06:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- 172.21.15.32:0/4060739242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 msgr2=0x7f8e6c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- 172.21.15.32:0/4060739242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c070360 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c06c080 tx=0x7f8e5c01c390 comp rx=0 tx=0).stop 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- 172.21.15.32:0/4060739242 shutdown_connections 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- 172.21.15.32:0/4060739242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c070360 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- 172.21.15.32:0/4060739242 >> v2:172.21.15.59:3300/0 conn(0x7f8e6c150d70 0x7f8e6c073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- 172.21.15.32:0/4060739242 >> 172.21.15.32:0/4060739242 conn(0x7f8e6c06c0d0 msgr2=0x7f8e6c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- 172.21.15.32:0/4060739242 shutdown_connections 2024-03-16T20:06:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- 172.21.15.32:0/4060739242 wait complete. 2024-03-16T20:06:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 Processor -- start 2024-03-16T20:06:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- start start 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8e6c0708a0 0x7f8e6c1f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6c150d70 0x7f8e6c149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c148ea0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1f6e20 con 0x7f8e6c150d70 2024-03-16T20:06:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.560+0000 7f8e73c47640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1f7100 con 0x7f8e6c0708a0 2024-03-16T20:06:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.564+0000 7f8e711e2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.564+0000 7f8e711e2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55036/0 (socket says 172.21.15.32:55036) 2024-03-16T20:06:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.564+0000 7f8e711e2640 1 -- 172.21.15.32:0/3327285789 learned_addr learned my addr 172.21.15.32:0/3327285789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.564+0000 7f8e719e3640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8e6c0708a0 0x7f8e6c1f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.564+0000 7f8e721e4640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6c150d70 0x7f8e6c149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.568+0000 7f8e711e2640 1 -- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8e6c0708a0 msgr2=0x7f8e6c1f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.568+0000 7f8e711e2640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8e6c0708a0 0x7f8e6c1f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.568+0000 7f8e711e2640 1 -- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6c150d70 msgr2=0x7f8e6c149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.568+0000 7f8e711e2640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6c150d70 0x7f8e6c149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.568+0000 7f8e711e2640 1 -- 172.21.15.32:0/3327285789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0094d0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.572+0000 7f8e711e2640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c019fc0 tx=0x7f8e5c019130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.576+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0b77f0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.576+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c0a6040 con 0x7f8e6c0725e0 2024-03-16T20:06:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.576+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c15d560 con 0x7f8e6c0725e0 2024-03-16T20:06:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.576+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0b7b90 con 0x7f8e6c0725e0 2024-03-16T20:06:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.576+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c150200 con 0x7f8e6c0725e0 2024-03-16T20:06:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.580+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c068000 con 0x7f8e6c0725e0 2024-03-16T20:06:08.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.920+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8e5c0b7cf0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.920+0000 7f8e627fc640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 0x7f8e50069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.920+0000 7f8e719e3640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 0x7f8e50069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.920+0000 7f8e719e3640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 0x7f8e50069350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8e68007a60 tx=0x7f8e68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.944+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8e5c0ad5c0 con 0x7f8e6c0725e0 2024-03-16T20:06:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:08.944+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8e5c0ece90 con 0x7f8e6c0725e0 2024-03-16T20:06:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.160+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6c068000 con 0x7f8e6c0725e0 2024-03-16T20:06:09.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.188+0000 7f8e627fc640 1 -- 172.21.15.32:0/3327285789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f8e5c0b2020 con 0x7f8e6c0725e0 2024-03-16T20:06:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:09.173 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:09.173 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 msgr2=0x7f8e50069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 0x7f8e50069350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8e68007a60 tx=0x7f8e68008040 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 msgr2=0x7f8e6c149350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c019fc0 tx=0x7f8e5c019130 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 shutdown_connections 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8e50066e90 0x7f8e50069350 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6c150d70 0x7f8e6c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e6c0725e0 0x7f8e6c149350 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 --2- 172.21.15.32:0/3327285789 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8e6c0708a0 0x7f8e6c1f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 >> 172.21.15.32:0/3327285789 conn(0x7f8e6c06c0d0 msgr2=0x7f8e6c0c7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 shutdown_connections 2024-03-16T20:06:09.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:09.192+0000 7f8e73c47640 1 -- 172.21.15.32:0/3327285789 wait complete. 2024-03-16T20:06:09.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:09.802 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:10.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 Processor -- start 2024-03-16T20:06:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 -- start start 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83e0151ef0 0x7f83e00c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83e00c97b0 0x7f83e00c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e00c8720 0x7f83e015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e00b0ad0 con 0x7f83e0151ef0 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e0157cb0 con 0x7f83e00c97b0 2024-03-16T20:06:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.924+0000 7f83e81ab640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0158c80 con 0x7f83e00c8720 2024-03-16T20:06:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83e0151ef0 0x7f83e00c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83e0151ef0 0x7f83e00c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41314/0 (socket says 172.21.15.32:41314) 2024-03-16T20:06:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 -- 172.21.15.32:0/3849965781 learned_addr learned my addr 172.21.15.32:0/3849965781 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5746640 1 --2- 172.21.15.32:0/3849965781 >> v2:172.21.15.59:3300/0 conn(0x7f83e00c97b0 0x7f83e00c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e6748640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e00c8720 0x7f83e015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 -- 172.21.15.32:0/3849965781 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e00c8720 msgr2=0x7f83e015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e00c8720 0x7f83e015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 -- 172.21.15.32:0/3849965781 >> v2:172.21.15.59:3300/0 conn(0x7f83e00c97b0 msgr2=0x7f83e00c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 --2- 172.21.15.32:0/3849965781 >> v2:172.21.15.59:3300/0 conn(0x7f83e00c97b0 0x7f83e00c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.928+0000 7f83e5f47640 1 -- 172.21.15.32:0/3849965781 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e015bfc0 con 0x7f83e0151ef0 2024-03-16T20:06:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.932+0000 7f83e5f47640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e0151ef0 0x7f83e00c9270 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f83e00cad10 tx=0x7f83dc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81072fecf99c2fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.936+0000 7f83e4f45640 1 -- 172.21.15.32:0/3849965781 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc0a89d0 con 0x7f83e0151ef0 2024-03-16T20:06:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e4f45640 1 -- 172.21.15.32:0/3849965781 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc0a8b30 con 0x7f83e0151ef0 2024-03-16T20:06:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e4f45640 1 -- 172.21.15.32:0/3849965781 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc0b0440 con 0x7f83e0151ef0 2024-03-16T20:06:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- 172.21.15.32:0/3849965781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e0151ef0 msgr2=0x7f83e00c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e0151ef0 0x7f83e00c9270 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f83e00cad10 tx=0x7f83dc01ed10 comp rx=0 tx=0).stop 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- 172.21.15.32:0/3849965781 shutdown_connections 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e00c8720 0x7f83e015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- 172.21.15.32:0/3849965781 >> v2:172.21.15.59:3300/0 conn(0x7f83e00c97b0 0x7f83e00c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- 172.21.15.32:0/3849965781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e0151ef0 0x7f83e00c9270 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- 172.21.15.32:0/3849965781 >> 172.21.15.32:0/3849965781 conn(0x7f83e0070f90 msgr2=0x7f83e00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- 172.21.15.32:0/3849965781 shutdown_connections 2024-03-16T20:06:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- 172.21.15.32:0/3849965781 wait complete. 2024-03-16T20:06:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 Processor -- start 2024-03-16T20:06:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- start start 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83e00c8720 0x7f83e01f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e0151ef0 0x7f83e015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e01f1f00 con 0x7f83e00c97b0 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e014d8f0 con 0x7f83e00c8720 2024-03-16T20:06:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.940+0000 7f83e81ab640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e014dbd0 con 0x7f83e0151ef0 2024-03-16T20:06:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.944+0000 7f83e5746640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.944+0000 7f83e5f47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83e00c8720 0x7f83e01f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.944+0000 7f83e5746640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41330/0 (socket says 172.21.15.32:41330) 2024-03-16T20:06:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.944+0000 7f83e5746640 1 -- 172.21.15.32:0/649984413 learned_addr learned my addr 172.21.15.32:0/649984413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.944+0000 7f83e6748640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e0151ef0 0x7f83e015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.948+0000 7f83e5746640 1 -- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e0151ef0 msgr2=0x7f83e015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.948+0000 7f83e5746640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e0151ef0 0x7f83e015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.948+0000 7f83e5746640 1 -- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83e00c8720 msgr2=0x7f83e01f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.948+0000 7f83e5746640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83e00c8720 0x7f83e01f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.948+0000 7f83e5746640 1 -- 172.21.15.32:0/649984413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc0094d0 con 0x7f83e00c97b0 2024-03-16T20:06:10.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.952+0000 7f83e5746640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d800 tx=0x7f83d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.956+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d000b6c0 con 0x7f83e00c97b0 2024-03-16T20:06:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.956+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d000b840 con 0x7f83e00c97b0 2024-03-16T20:06:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.956+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e014deb0 con 0x7f83e00c97b0 2024-03-16T20:06:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.956+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d000bbf0 con 0x7f83e00c97b0 2024-03-16T20:06:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.956+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e01f14a0 con 0x7f83e00c97b0 2024-03-16T20:06:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:10.960+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e01f1020 con 0x7f83e00c97b0 2024-03-16T20:06:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.252+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f83d000bd50 con 0x7f83e00c97b0 2024-03-16T20:06:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.252+0000 7f83d67fc640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 0x7f83c4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.252+0000 7f83e5f47640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 0x7f83c4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.252+0000 7f83e5f47640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 0x7f83c4069380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0b2000 tx=0x7f83dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:11.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.276+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f83d0056350 con 0x7f83e00c97b0 2024-03-16T20:06:11.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.276+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f83d00525f0 con 0x7f83e00c97b0 2024-03-16T20:06:11.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.480+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83e01f1020 con 0x7f83e00c97b0 2024-03-16T20:06:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83d67fc640 1 -- 172.21.15.32:0/649984413 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f83e01f1020 con 0x7f83e00c97b0 2024-03-16T20:06:11.552 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:11.553 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 msgr2=0x7f83c4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 0x7f83c4069380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0b2000 tx=0x7f83dc009040 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 msgr2=0x7f83e00c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d800 tx=0x7f83d000dcc0 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 shutdown_connections 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f83c4066ec0 0x7f83c4069380 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f83e0151ef0 0x7f83e015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83e00c97b0 0x7f83e00c9e20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 --2- 172.21.15.32:0/649984413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83e00c8720 0x7f83e01f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 >> 172.21.15.32:0/649984413 conn(0x7f83e0070f90 msgr2=0x7f83e006efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 shutdown_connections 2024-03-16T20:06:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:11.568+0000 7f83e81ab640 1 -- 172.21.15.32:0/649984413 wait complete. 2024-03-16T20:06:12.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:12.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:12.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:06:12.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:06:13.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:13.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 Processor -- start 2024-03-16T20:06:13.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 -- start start 2024-03-16T20:06:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff63c0725e0 0x7ff63c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c151020 con 0x7ff63c15a420 2024-03-16T20:06:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c072da0 con 0x7ff63c150c40 2024-03-16T20:06:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff642275640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c15a000 con 0x7ff63c0725e0 2024-03-16T20:06:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff63b7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff63c0725e0 0x7ff63c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:13.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff63bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff63c150c40 0x7ff63c070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49234/0 (socket says 172.21.15.32:49234) 2024-03-16T20:06:13.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.307+0000 7ff63bfff640 1 -- 172.21.15.32:0/1697596941 learned_addr learned my addr 172.21.15.32:0/1697596941 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.315+0000 7ff63b7fe640 1 -- 172.21.15.32:0/1697596941 >> v1:172.21.15.59:6789/0 conn(0x7ff63c150c40 legacy=0x7ff63c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.315+0000 7ff63b7fe640 1 -- 172.21.15.32:0/1697596941 >> v1:172.21.15.32:6789/0 conn(0x7ff63c15a420 legacy=0x7ff63c15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.315+0000 7ff63b7fe640 1 -- 172.21.15.32:0/1697596941 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c15e310 con 0x7ff63c0725e0 2024-03-16T20:06:13.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.323+0000 7ff63b7fe640 1 --2- 172.21.15.32:0/1697596941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c073070 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6300097c0 tx=0x7ff63001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea374aadff59849 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:13.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff63affd640 1 -- 172.21.15.32:0/1697596941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300a6040 con 0x7ff63c0725e0 2024-03-16T20:06:13.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff63affd640 1 -- 172.21.15.32:0/1697596941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300aa600 con 0x7ff63c0725e0 2024-03-16T20:06:13.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff63affd640 1 -- 172.21.15.32:0/1697596941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300aa990 con 0x7ff63c0725e0 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 -- 172.21.15.32:0/1697596941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 msgr2=0x7ff63c073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 --2- 172.21.15.32:0/1697596941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c073070 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6300097c0 tx=0x7ff63001c4a0 comp rx=0 tx=0).stop 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 -- 172.21.15.32:0/1697596941 shutdown_connections 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 --2- 172.21.15.32:0/1697596941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c073070 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 -- 172.21.15.32:0/1697596941 >> 172.21.15.32:0/1697596941 conn(0x7ff63c06c0d0 msgr2=0x7ff63c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 -- 172.21.15.32:0/1697596941 shutdown_connections 2024-03-16T20:06:13.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.327+0000 7ff642275640 1 -- 172.21.15.32:0/1697596941 wait complete. 2024-03-16T20:06:13.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 Processor -- start 2024-03-16T20:06:13.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 -- start start 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff63c150c40 0x7ff63c142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff63c15a420 0x7ff63c1434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1f3510 con 0x7ff63c0725e0 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1f2640 con 0x7ff63c150c40 2024-03-16T20:06:13.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff642275640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1f2900 con 0x7ff63c15a420 2024-03-16T20:06:13.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:13.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff63b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff63c150c40 0x7ff63c142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:13.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41346/0 (socket says 172.21.15.32:41346) 2024-03-16T20:06:13.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff63bfff640 1 -- 172.21.15.32:0/2636096272 learned_addr learned my addr 172.21.15.32:0/2636096272 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:13.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.331+0000 7ff640812640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff63c15a420 0x7ff63c1434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.335+0000 7ff63bfff640 1 -- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff63c15a420 msgr2=0x7ff63c1434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.335+0000 7ff63bfff640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff63c15a420 0x7ff63c1434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.335+0000 7ff63bfff640 1 -- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff63c150c40 msgr2=0x7ff63c142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.335+0000 7ff63bfff640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff63c150c40 0x7ff63c142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.335+0000 7ff63bfff640 1 -- 172.21.15.32:0/2636096272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009420 con 0x7ff63c0725e0 2024-03-16T20:06:13.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.339+0000 7ff63bfff640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c500 tx=0x7ff62c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:13.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c005cf0 con 0x7ff63c0725e0 2024-03-16T20:06:13.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c005e50 con 0x7ff63c0725e0 2024-03-16T20:06:13.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0072f0 con 0x7ff63c0725e0 2024-03-16T20:06:13.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c15d560 con 0x7ff63c0725e0 2024-03-16T20:06:13.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c1f2ec0 con 0x7ff63c0725e0 2024-03-16T20:06:13.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.343+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c068000 con 0x7ff63c0725e0 2024-03-16T20:06:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.691+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff62c007450 con 0x7ff63c0725e0 2024-03-16T20:06:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.691+0000 7ff638ff9640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 0x7ff628069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.691+0000 7ff63b7fe640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 0x7ff628069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.691+0000 7ff63b7fe640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 0x7ff628069300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ae000 tx=0x7ff6300b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:13.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.711+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff62c0550b0 con 0x7ff63c0725e0 2024-03-16T20:06:13.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.711+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff62c051350 con 0x7ff63c0725e0 2024-03-16T20:06:13.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:13.995+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff63c0cc3f0 con 0x7ff63c0725e0 2024-03-16T20:06:13.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.011+0000 7ff638ff9640 1 -- 172.21.15.32:0/2636096272 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff62c017020 con 0x7ff63c0725e0 2024-03-16T20:06:13.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:13.998 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:13.998 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 msgr2=0x7ff628069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 0x7ff628069300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ae000 tx=0x7ff6300b5040 comp rx=0 tx=0).stop 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 msgr2=0x7ff63c0c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c500 tx=0x7ff62c00c9c0 comp rx=0 tx=0).stop 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 shutdown_connections 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff628066e40 0x7ff628069300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff63c15a420 0x7ff63c1434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff63c150c40 0x7ff63c142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 --2- 172.21.15.32:0/2636096272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c0725e0 0x7ff63c0c62f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 >> 172.21.15.32:0/2636096272 conn(0x7ff63c06c0d0 msgr2=0x7ff63c0c8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 shutdown_connections 2024-03-16T20:06:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:14.015+0000 7ff642275640 1 -- 172.21.15.32:0/2636096272 wait complete. 2024-03-16T20:06:14.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:14.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:15.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:15.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.747+0000 7fe48f4ac640 1 Processor -- start 2024-03-16T20:06:15.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.747+0000 7fe48f4ac640 1 -- start start 2024-03-16T20:06:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48f4ac640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4880c81e0 0x7fe4880c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48f4ac640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 0x7fe48815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48f4ac640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe488158c80 con 0x7fe488151ef0 2024-03-16T20:06:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48f4ac640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4880b0ad0 con 0x7fe4880c81e0 2024-03-16T20:06:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48f4ac640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe488157cb0 con 0x7fe48815b2a0 2024-03-16T20:06:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48ca47640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4880c81e0 0x7fe4880c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48d248640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe488151ef0 0x7fe4880c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41062/0 (socket says 172.21.15.32:41062) 2024-03-16T20:06:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48d248640 1 -- 172.21.15.32:0/3301992348 learned_addr learned my addr 172.21.15.32:0/3301992348 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.751+0000 7fe48da49640 1 --2- 172.21.15.32:0/3301992348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 0x7fe48815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:15.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.759+0000 7fe48da49640 1 -- 172.21.15.32:0/3301992348 >> v2:172.21.15.32:3300/0 conn(0x7fe4880c81e0 msgr2=0x7fe4880c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:15.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.759+0000 7fe48da49640 1 --2- 172.21.15.32:0/3301992348 >> v2:172.21.15.32:3300/0 conn(0x7fe4880c81e0 0x7fe4880c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:15.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.759+0000 7fe48da49640 1 -- 172.21.15.32:0/3301992348 >> v1:172.21.15.32:6789/0 conn(0x7fe488151ef0 legacy=0x7fe4880c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:15.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.759+0000 7fe48da49640 1 -- 172.21.15.32:0/3301992348 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48815bfc0 con 0x7fe48815b2a0 2024-03-16T20:06:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.763+0000 7fe48da49640 1 --2- 172.21.15.32:0/3301992348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 0x7fe48815b6a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe480009d90 tx=0x7fe48001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d6824c0cb03a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:15.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.771+0000 7fe47ffff640 1 -- 172.21.15.32:0/3301992348 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4800af600 con 0x7fe48815b2a0 2024-03-16T20:06:15.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.771+0000 7fe47ffff640 1 -- 172.21.15.32:0/3301992348 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4800ab050 con 0x7fe48815b2a0 2024-03-16T20:06:15.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.771+0000 7fe47ffff640 1 -- 172.21.15.32:0/3301992348 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4800af980 con 0x7fe48815b2a0 2024-03-16T20:06:15.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3301992348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 msgr2=0x7fe48815b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:15.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3301992348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 0x7fe48815b6a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe480009d90 tx=0x7fe48001f150 comp rx=0 tx=0).stop 2024-03-16T20:06:15.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3301992348 shutdown_connections 2024-03-16T20:06:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3301992348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe48815b2a0 0x7fe48815b6a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3301992348 >> v2:172.21.15.32:3300/0 conn(0x7fe4880c81e0 0x7fe4880c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3301992348 >> 172.21.15.32:0/3301992348 conn(0x7fe488070f90 msgr2=0x7fe4880713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3301992348 shutdown_connections 2024-03-16T20:06:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3301992348 wait complete. 2024-03-16T20:06:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 Processor -- start 2024-03-16T20:06:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 -- start start 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe488151ef0 0x7fe48814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.775+0000 7fe48f4ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe48815b2a0 0x7fe48814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48f4ac640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488150990 con 0x7fe4880c81e0 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48f4ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488150c00 con 0x7fe48815b2a0 2024-03-16T20:06:15.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48f4ac640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe488150ee0 con 0x7fe488151ef0 2024-03-16T20:06:15.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48d248640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:15.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48d248640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41350/0 (socket says 172.21.15.32:41350) 2024-03-16T20:06:15.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48d248640 1 -- 172.21.15.32:0/3140571034 learned_addr learned my addr 172.21.15.32:0/3140571034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:15.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48ca47640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe488151ef0 0x7fe48814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:15.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.779+0000 7fe48da49640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe48815b2a0 0x7fe48814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.783+0000 7fe48d248640 1 -- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe488151ef0 msgr2=0x7fe48814f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.783+0000 7fe48d248640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe488151ef0 0x7fe48814f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.783+0000 7fe48d248640 1 -- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe48815b2a0 msgr2=0x7fe48814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.783+0000 7fe48d248640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe48815b2a0 0x7fe48814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.783+0000 7fe48d248640 1 -- 172.21.15.32:0/3140571034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe480009a70 con 0x7fe4880c81e0 2024-03-16T20:06:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.787+0000 7fe48d248640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c420 tx=0x7fe48400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:15.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.791+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484005cf0 con 0x7fe4880c81e0 2024-03-16T20:06:15.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.791+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4881511c0 con 0x7fe4880c81e0 2024-03-16T20:06:15.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.791+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4880b2fe0 con 0x7fe4880c81e0 2024-03-16T20:06:15.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.795+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484005e50 con 0x7fe4880c81e0 2024-03-16T20:06:15.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.795+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840072d0 con 0x7fe4880c81e0 2024-03-16T20:06:15.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:15.795+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4680052a0 con 0x7fe4880c81e0 2024-03-16T20:06:16.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.087+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe484007430 con 0x7fe4880c81e0 2024-03-16T20:06:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.087+0000 7fe47dffb640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 0x7fe46c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.087+0000 7fe48ca47640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 0x7fe46c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.087+0000 7fe48ca47640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 0x7fe46c069330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe4880b0aa0 tx=0x7fe478009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:16.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.131+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe484021c80 con 0x7fe4880c81e0 2024-03-16T20:06:16.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.139+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe4840515c0 con 0x7fe4880c81e0 2024-03-16T20:06:16.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.359+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe468005900 con 0x7fe4880c81e0 2024-03-16T20:06:16.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe47dffb640 1 -- 172.21.15.32:0/3140571034 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fe468005900 con 0x7fe4880c81e0 2024-03-16T20:06:16.367 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:16.368 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:16.368 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 msgr2=0x7fe46c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 0x7fe46c069330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe4880b0aa0 tx=0x7fe478009190 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 msgr2=0x7fe4880c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c420 tx=0x7fe48400c8e0 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 shutdown_connections 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe46c066e70 0x7fe46c069330 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe48815b2a0 0x7fe48814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe488151ef0 0x7fe48814f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 --2- 172.21.15.32:0/3140571034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c81e0 0x7fe4880c6650 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 >> 172.21.15.32:0/3140571034 conn(0x7fe488070f90 msgr2=0x7fe4880ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 shutdown_connections 2024-03-16T20:06:16.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:16.383+0000 7fe48f4ac640 1 -- 172.21.15.32:0/3140571034 wait complete. 2024-03-16T20:06:16.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:16.990 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:17.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:18.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.111+0000 7f56f166b640 1 Processor -- start 2024-03-16T20:06:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.111+0000 7f56f166b640 1 -- start start 2024-03-16T20:06:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56f166b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56ec0725e0 0x7f56ec073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56f166b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56ec15a420 0x7f56ec15a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56f166b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec151020 con 0x7f56ec150c40 2024-03-16T20:06:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56f166b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec15a000 con 0x7f56ec15a420 2024-03-16T20:06:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56f166b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec15add0 con 0x7f56ec0725e0 2024-03-16T20:06:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56ea7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56ec0725e0 0x7f56ec073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56eb7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56ec15a420 0x7f56ec15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56ea7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56ec0725e0 0x7f56ec073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:51688/0 (socket says 172.21.15.32:51688) 2024-03-16T20:06:18.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.115+0000 7f56ea7fc640 1 -- 172.21.15.32:0/1733260174 learned_addr learned my addr 172.21.15.32:0/1733260174 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:18.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.119+0000 7f56ea7fc640 1 -- 172.21.15.32:0/1733260174 >> v2:172.21.15.32:3300/0 conn(0x7f56ec15a420 msgr2=0x7f56ec15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:18.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.119+0000 7f56ea7fc640 1 --2- 172.21.15.32:0/1733260174 >> v2:172.21.15.32:3300/0 conn(0x7f56ec15a420 0x7f56ec15a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.119+0000 7f56ea7fc640 1 -- 172.21.15.32:0/1733260174 >> v1:172.21.15.32:6789/0 conn(0x7f56ec150c40 legacy=0x7f56ec070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.119+0000 7f56ea7fc640 1 -- 172.21.15.32:0/1733260174 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec072da0 con 0x7f56ec0725e0 2024-03-16T20:06:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.123+0000 7f56ea7fc640 1 --2- 172.21.15.32:0/1733260174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec073070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f56ec06fba0 tx=0x7f56e001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5342bc0937aee34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.127+0000 7f56e9ffb640 1 -- 172.21.15.32:0/1733260174 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00a9400 con 0x7f56ec0725e0 2024-03-16T20:06:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.127+0000 7f56e9ffb640 1 -- 172.21.15.32:0/1733260174 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e00a9560 con 0x7f56ec0725e0 2024-03-16T20:06:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.127+0000 7f56e9ffb640 1 -- 172.21.15.32:0/1733260174 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00a9930 con 0x7f56ec0725e0 2024-03-16T20:06:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- 172.21.15.32:0/1733260174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 msgr2=0x7f56ec073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 --2- 172.21.15.32:0/1733260174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec073070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f56ec06fba0 tx=0x7f56e001c540 comp rx=0 tx=0).stop 2024-03-16T20:06:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- 172.21.15.32:0/1733260174 shutdown_connections 2024-03-16T20:06:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 --2- 172.21.15.32:0/1733260174 >> v2:172.21.15.32:3300/0 conn(0x7f56ec15a420 0x7f56ec15a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 --2- 172.21.15.32:0/1733260174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec073070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- 172.21.15.32:0/1733260174 >> 172.21.15.32:0/1733260174 conn(0x7f56ec06c0d0 msgr2=0x7f56ec06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- 172.21.15.32:0/1733260174 shutdown_connections 2024-03-16T20:06:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- 172.21.15.32:0/1733260174 wait complete. 2024-03-16T20:06:18.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 Processor -- start 2024-03-16T20:06:18.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.131+0000 7f56f166b640 1 -- start start 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec14d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56ec150c40 0x7f56ec1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 0x7f56ec0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f6490 con 0x7f56ec15a420 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f6730 con 0x7f56ec0725e0 2024-03-16T20:06:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56f166b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f6a10 con 0x7f56ec150c40 2024-03-16T20:06:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec14d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56eb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 0x7f56ec0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec14d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:51696/0 (socket says 172.21.15.32:51696) 2024-03-16T20:06:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56eaffd640 1 -- 172.21.15.32:0/1384299836 learned_addr learned my addr 172.21.15.32:0/1384299836 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.135+0000 7f56ea7fc640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56ec150c40 0x7f56ec1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.139+0000 7f56eb7fe640 1 -- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56ec150c40 msgr2=0x7f56ec1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.139+0000 7f56eb7fe640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56ec150c40 0x7f56ec1499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.139+0000 7f56eb7fe640 1 -- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 msgr2=0x7f56ec14d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.139+0000 7f56eb7fe640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec14d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.139+0000 7f56eb7fe640 1 -- 172.21.15.32:0/1384299836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00094d0 con 0x7f56ec15a420 2024-03-16T20:06:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.143+0000 7f56eb7fe640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 0x7f56ec0cc910 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f56e400eba0 tx=0x7f56e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:18.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400cd20 con 0x7f56ec15a420 2024-03-16T20:06:18.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e400f040 con 0x7f56ec15a420 2024-03-16T20:06:18.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec1f6cf0 con 0x7f56ec15a420 2024-03-16T20:06:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40146c0 con 0x7f56ec15a420 2024-03-16T20:06:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec14a090 con 0x7f56ec15a420 2024-03-16T20:06:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.147+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec1f7210 con 0x7f56ec15a420 2024-03-16T20:06:18.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.179+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f56e4014820 con 0x7f56ec15a420 2024-03-16T20:06:18.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.179+0000 7f56d37fe640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 0x7f56d8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:18.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.179+0000 7f56eaffd640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 0x7f56d8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.183+0000 7f56eaffd640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 0x7f56d8069330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f56dc007fd0 tx=0x7f56dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.487+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f56e4056d50 con 0x7f56ec15a420 2024-03-16T20:06:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.487+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f56e40528f0 con 0x7f56ec15a420 2024-03-16T20:06:18.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.699+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56ec1f7210 con 0x7f56ec15a420 2024-03-16T20:06:18.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.719+0000 7f56d37fe640 1 -- 172.21.15.32:0/1384299836 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f56ec1f7210 con 0x7f56ec15a420 2024-03-16T20:06:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:18.707 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:18.707 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 msgr2=0x7f56d8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 0x7f56d8069330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f56dc007fd0 tx=0x7f56dc009040 comp rx=0 tx=0).stop 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 msgr2=0x7f56ec0cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 0x7f56ec0cc910 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f56e400eba0 tx=0x7f56e400c490 comp rx=0 tx=0).stop 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 shutdown_connections 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f56d8066e70 0x7f56d8069330 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56ec15a420 0x7f56ec0cc910 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56ec150c40 0x7f56ec1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 --2- 172.21.15.32:0/1384299836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56ec0725e0 0x7f56ec14d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:18.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 >> 172.21.15.32:0/1384299836 conn(0x7f56ec06c0d0 msgr2=0x7f56ec0c8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:18.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 shutdown_connections 2024-03-16T20:06:18.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:18.723+0000 7f56f166b640 1 -- 172.21.15.32:0/1384299836 wait complete. 2024-03-16T20:06:19.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:19.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:20.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:20.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.459+0000 7ff9ee2ea640 1 Processor -- start 2024-03-16T20:06:20.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.459+0000 7ff9ee2ea640 1 -- start start 2024-03-16T20:06:20.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ee2ea640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9e806f220 0x7ff9e815d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ee2ea640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff9e806f600 0x7ff9e815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ee2ea640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e8159f10 con 0x7ff9e80c9ec0 2024-03-16T20:06:20.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ee2ea640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e815a1d0 con 0x7ff9e806f220 2024-03-16T20:06:20.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ee2ea640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e815a4b0 con 0x7ff9e806f600 2024-03-16T20:06:20.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9e7fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9e806f220 0x7ff9e815d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9e77fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff9e80c9ec0 0x7ff9e8153cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35006/0 (socket says 172.21.15.32:35006) 2024-03-16T20:06:20.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9e77fe640 1 -- 172.21.15.32:0/3347443699 learned_addr learned my addr 172.21.15.32:0/3347443699 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:20.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.463+0000 7ff9ec887640 1 --2- 172.21.15.32:0/3347443699 >> v2:172.21.15.59:3300/0 conn(0x7ff9e806f600 0x7ff9e815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.467+0000 7ff9e6ffd640 1 -- 172.21.15.32:0/3347443699 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908091569 0 0) 0x7ff9e8159f10 con 0x7ff9e80c9ec0 2024-03-16T20:06:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.467+0000 7ff9e6ffd640 1 -- 172.21.15.32:0/3347443699 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d0003530 con 0x7ff9e80c9ec0 2024-03-16T20:06:20.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.471+0000 7ff9ec887640 1 -- 172.21.15.32:0/3347443699 >> v2:172.21.15.32:3300/0 conn(0x7ff9e806f220 msgr2=0x7ff9e815d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:20.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.471+0000 7ff9ec887640 1 --2- 172.21.15.32:0/3347443699 >> v2:172.21.15.32:3300/0 conn(0x7ff9e806f220 0x7ff9e815d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.471+0000 7ff9ec887640 1 -- 172.21.15.32:0/3347443699 >> v1:172.21.15.32:6789/0 conn(0x7ff9e80c9ec0 legacy=0x7ff9e8153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:20.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.471+0000 7ff9ec887640 1 -- 172.21.15.32:0/3347443699 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e815a790 con 0x7ff9e806f600 2024-03-16T20:06:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.475+0000 7ff9ec887640 1 --2- 172.21.15.32:0/3347443699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f600 0x7ff9e815dbe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff9e806ce80 tx=0x7ff9e001c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4baec2df4e3dc5b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9e6ffd640 1 -- 172.21.15.32:0/3347443699 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00a5960 con 0x7ff9e806f600 2024-03-16T20:06:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9e6ffd640 1 -- 172.21.15.32:0/3347443699 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00a5ac0 con 0x7ff9e806f600 2024-03-16T20:06:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9e6ffd640 1 -- 172.21.15.32:0/3347443699 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00a5e50 con 0x7ff9e806f600 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/3347443699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f600 msgr2=0x7ff9e815dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/3347443699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f600 0x7ff9e815dbe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff9e806ce80 tx=0x7ff9e001c8a0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/3347443699 shutdown_connections 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/3347443699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f600 0x7ff9e815dbe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/3347443699 >> v2:172.21.15.32:3300/0 conn(0x7ff9e806f220 0x7ff9e815d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/3347443699 >> 172.21.15.32:0/3347443699 conn(0x7ff9e806b1c0 msgr2=0x7ff9e8066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/3347443699 shutdown_connections 2024-03-16T20:06:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/3347443699 wait complete. 2024-03-16T20:06:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 Processor -- start 2024-03-16T20:06:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.487+0000 7ff9ee2ea640 1 -- start start 2024-03-16T20:06:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f220 0x7ff9e815c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e80c9ec0 0x7ff9e80cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80cba90 con 0x7ff9e806f600 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8157d80 con 0x7ff9e806f220 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ee2ea640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81582b0 con 0x7ff9e80c9ec0 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49942/0 (socket says 172.21.15.32:49942) 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9e77fe640 1 -- 172.21.15.32:0/2580025911 learned_addr learned my addr 172.21.15.32:0/2580025911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9ec887640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e80c9ec0 0x7ff9e80cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.491+0000 7ff9e7fff640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f220 0x7ff9e815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.495+0000 7ff9e77fe640 1 -- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e80c9ec0 msgr2=0x7ff9e80cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.495+0000 7ff9e77fe640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e80c9ec0 0x7ff9e80cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.495+0000 7ff9e77fe640 1 -- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f220 msgr2=0x7ff9e815c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.495+0000 7ff9e77fe640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f220 0x7ff9e815c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.495+0000 7ff9e77fe640 1 -- 172.21.15.32:0/2580025911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0009a70 con 0x7ff9e806f600 2024-03-16T20:06:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.499+0000 7ff9e77fe640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c480 tx=0x7ff9dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:20.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc005cf0 con 0x7ff9e806f600 2024-03-16T20:06:20.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc005e50 con 0x7ff9e806f600 2024-03-16T20:06:20.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8157fd0 con 0x7ff9e806f600 2024-03-16T20:06:20.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc007290 con 0x7ff9e806f600 2024-03-16T20:06:20.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81599e0 con 0x7ff9e806f600 2024-03-16T20:06:20.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.507+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8068000 con 0x7ff9e806f600 2024-03-16T20:06:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.543+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff9dc0073f0 con 0x7ff9e806f600 2024-03-16T20:06:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.543+0000 7ff9e4ff9640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 0x7ff9d4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.543+0000 7ff9e7fff640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 0x7ff9d4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.543+0000 7ff9e7fff640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 0x7ff9d4069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009ac0 tx=0x7ff9d8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:20.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.847+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff9dc0550c0 con 0x7ff9e806f600 2024-03-16T20:06:20.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:20.847+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff9dc051360 con 0x7ff9e806f600 2024-03-16T20:06:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.095+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e80c86f0 con 0x7ff9e806f600 2024-03-16T20:06:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.123+0000 7ff9e4ff9640 1 -- 172.21.15.32:0/2580025911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7ff9dc017090 con 0x7ff9e806f600 2024-03-16T20:06:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:21.110 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 msgr2=0x7ff9d4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 0x7ff9d4069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009ac0 tx=0x7ff9d8009190 comp rx=0 tx=0).stop 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 msgr2=0x7ff9e80cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c480 tx=0x7ff9dc00c940 comp rx=0 tx=0).stop 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 shutdown_connections 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9d4066e90 0x7ff9d4069350 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e80c9ec0 0x7ff9e80cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e806f600 0x7ff9e80cccf0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 --2- 172.21.15.32:0/2580025911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e806f220 0x7ff9e815c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 >> 172.21.15.32:0/2580025911 conn(0x7ff9e806b1c0 msgr2=0x7ff9e814e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 shutdown_connections 2024-03-16T20:06:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:21.127+0000 7ff9ee2ea640 1 -- 172.21.15.32:0/2580025911 wait complete. 2024-03-16T20:06:21.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:21.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:22.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:22.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.859+0000 7f5510a21640 1 Processor -- start 2024-03-16T20:06:22.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.859+0000 7f5510a21640 1 -- start start 2024-03-16T20:06:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5510a21640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f550c153fc0 0x7f550c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5510a21640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f550c0c8050 0x7f550c0c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5510a21640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c15ad50 con 0x7f550c0c72f0 2024-03-16T20:06:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5510a21640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550c0c7ce0 con 0x7f550c0c8050 2024-03-16T20:06:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5510a21640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550c15a1f0 con 0x7f550c153fc0 2024-03-16T20:06:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f550ad9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f550c0c8050 0x7f550c0c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5509d9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f550c0c72f0 0x7f550c15d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35020/0 (socket says 172.21.15.32:35020) 2024-03-16T20:06:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f5509d9b640 1 -- 172.21.15.32:0/3188695484 learned_addr learned my addr 172.21.15.32:0/3188695484 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.863+0000 7f550a59c640 1 --2- 172.21.15.32:0/3188695484 >> v2:172.21.15.59:3300/0 conn(0x7f550c153fc0 0x7f550c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.867+0000 7f550ad9d640 1 -- 172.21.15.32:0/3188695484 >> v2:172.21.15.59:3300/0 conn(0x7f550c153fc0 msgr2=0x7f550c0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.867+0000 7f550ad9d640 1 --2- 172.21.15.32:0/3188695484 >> v2:172.21.15.59:3300/0 conn(0x7f550c153fc0 0x7f550c0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.867+0000 7f550ad9d640 1 -- 172.21.15.32:0/3188695484 >> v1:172.21.15.32:6789/0 conn(0x7f550c0c72f0 legacy=0x7f550c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.867+0000 7f550ad9d640 1 -- 172.21.15.32:0/3188695484 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c15a4b0 con 0x7f550c0c8050 2024-03-16T20:06:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.871+0000 7f550ad9d640 1 --2- 172.21.15.32:0/3188695484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c8050 0x7f550c0c84c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5504009d90 tx=0x7f550401f140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b214c15a1c201d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:22.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f550959a640 1 -- 172.21.15.32:0/3188695484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55040af600 con 0x7f550c0c8050 2024-03-16T20:06:22.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f550959a640 1 -- 172.21.15.32:0/3188695484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55040ab050 con 0x7f550c0c8050 2024-03-16T20:06:22.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f550959a640 1 -- 172.21.15.32:0/3188695484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55040af980 con 0x7f550c0c8050 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 -- 172.21.15.32:0/3188695484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c8050 msgr2=0x7f550c0c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 --2- 172.21.15.32:0/3188695484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c8050 0x7f550c0c84c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5504009d90 tx=0x7f550401f140 comp rx=0 tx=0).stop 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 -- 172.21.15.32:0/3188695484 shutdown_connections 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 --2- 172.21.15.32:0/3188695484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c8050 0x7f550c0c84c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 --2- 172.21.15.32:0/3188695484 >> v2:172.21.15.59:3300/0 conn(0x7f550c153fc0 0x7f550c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 -- 172.21.15.32:0/3188695484 >> 172.21.15.32:0/3188695484 conn(0x7f550c0cc7a0 msgr2=0x7f550c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.875+0000 7f5510a21640 1 -- 172.21.15.32:0/3188695484 shutdown_connections 2024-03-16T20:06:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 -- 172.21.15.32:0/3188695484 wait complete. 2024-03-16T20:06:22.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 Processor -- start 2024-03-16T20:06:22.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 -- start start 2024-03-16T20:06:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c0c8050 0x7f550c14f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f550c153fc0 0x7f550c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c151e20 con 0x7f550c0c72f0 2024-03-16T20:06:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c15bc20 con 0x7f550c0c8050 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f5510a21640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c0cbe50 con 0x7f550c153fc0 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.879+0000 7f550a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.883+0000 7f550a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49962/0 (socket says 172.21.15.32:49962) 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.883+0000 7f550a59c640 1 -- 172.21.15.32:0/2176719918 learned_addr learned my addr 172.21.15.32:0/2176719918 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.883+0000 7f5509d9b640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c0c8050 0x7f550c14f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.883+0000 7f550ad9d640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f550c153fc0 0x7f550c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.887+0000 7f550a59c640 1 -- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f550c153fc0 msgr2=0x7f550c0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:22.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.887+0000 7f550a59c640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f550c153fc0 0x7f550c0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:22.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.887+0000 7f550a59c640 1 -- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c0c8050 msgr2=0x7f550c14f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:22.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.887+0000 7f550a59c640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c0c8050 0x7f550c14f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:22.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.887+0000 7f550a59c640 1 -- 172.21.15.32:0/2176719918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5504009a70 con 0x7f550c0c72f0 2024-03-16T20:06:22.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.891+0000 7f550a59c640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00e930 tx=0x7f54fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:22.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.895+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc009690 con 0x7f550c0c72f0 2024-03-16T20:06:22.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.895+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc00ca00 con 0x7f550c0c72f0 2024-03-16T20:06:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.895+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c0cc0f0 con 0x7f550c0c72f0 2024-03-16T20:06:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.895+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00cd70 con 0x7f550c0c72f0 2024-03-16T20:06:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.895+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c15b380 con 0x7f550c0c72f0 2024-03-16T20:06:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.899+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c068000 con 0x7f550c0c72f0 2024-03-16T20:06:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.931+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f54fc013470 con 0x7f550c0c72f0 2024-03-16T20:06:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.931+0000 7f54faffd640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 0x7f54ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.931+0000 7f5509d9b640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 0x7f54ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:22.935+0000 7f5509d9b640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 0x7f54ec069350 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f550c1eda80 tx=0x7f5500007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.219+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f54fc0560e0 con 0x7f550c0c72f0 2024-03-16T20:06:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.223+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f54fc052380 con 0x7f550c0c72f0 2024-03-16T20:06:23.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.555+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f550c1efe50 con 0x7f550c0c72f0 2024-03-16T20:06:23.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f54faffd640 1 -- 172.21.15.32:0/2176719918 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f54fc021070 con 0x7f550c0c72f0 2024-03-16T20:06:23.565 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:23.565 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:23.566 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:23.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 msgr2=0x7f54ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:23.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 0x7f54ec069350 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f550c1eda80 tx=0x7f5500007e90 comp rx=0 tx=0).stop 2024-03-16T20:06:23.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 msgr2=0x7f550c14f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:23.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00e930 tx=0x7f54fc00edf0 comp rx=0 tx=0).stop 2024-03-16T20:06:23.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 shutdown_connections 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f54ec066e90 0x7f54ec069350 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f550c153fc0 0x7f550c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c0c8050 0x7f550c14f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 --2- 172.21.15.32:0/2176719918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f550c0c72f0 0x7f550c14f0f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.579+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 >> 172.21.15.32:0/2176719918 conn(0x7f550c0cc7a0 msgr2=0x7f550c0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.583+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 shutdown_connections 2024-03-16T20:06:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:23.583+0000 7f5510a21640 1 -- 172.21.15.32:0/2176719918 wait complete. 2024-03-16T20:06:24.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:24.188 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:25.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 Processor -- start 2024-03-16T20:06:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 -- start start 2024-03-16T20:06:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c58070840 0x7f7c580c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c58070c90 con 0x7f7c58153fa0 2024-03-16T20:06:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c580c8590 con 0x7f7c580724a0 2024-03-16T20:06:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.311+0000 7f7c5d8f9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c580c3f00 con 0x7f7c58070840 2024-03-16T20:06:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.315+0000 7f7c567fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c58070840 0x7f7c580c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.315+0000 7f7c567fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c58070840 0x7f7c580c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49966/0 (socket says 172.21.15.32:49966) 2024-03-16T20:06:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.315+0000 7f7c567fc640 1 -- 172.21.15.32:0/2858970003 learned_addr learned my addr 172.21.15.32:0/2858970003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.319+0000 7f7c567fc640 1 -- 172.21.15.32:0/2858970003 >> v1:172.21.15.59:6789/0 conn(0x7f7c580724a0 legacy=0x7f7c58164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.319+0000 7f7c567fc640 1 -- 172.21.15.32:0/2858970003 >> v1:172.21.15.32:6789/0 conn(0x7f7c58153fa0 legacy=0x7f7c5815d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.319+0000 7f7c567fc640 1 -- 172.21.15.32:0/2858970003 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58073590 con 0x7f7c58070840 2024-03-16T20:06:25.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.323+0000 7f7c567fc640 1 --2- 172.21.15.32:0/2858970003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c580c7470 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009870 tx=0x7f7c4c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b130897f0e89e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c55ffb640 1 -- 172.21.15.32:0/2858970003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0a8a50 con 0x7f7c58070840 2024-03-16T20:06:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c55ffb640 1 -- 172.21.15.32:0/2858970003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c0a8bb0 con 0x7f7c58070840 2024-03-16T20:06:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c55ffb640 1 -- 172.21.15.32:0/2858970003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0b0440 con 0x7f7c58070840 2024-03-16T20:06:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/2858970003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 msgr2=0x7f7c580c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/2858970003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c580c7470 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009870 tx=0x7f7c4c01ed50 comp rx=0 tx=0).stop 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/2858970003 shutdown_connections 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/2858970003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c580c7470 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/2858970003 >> 172.21.15.32:0/2858970003 conn(0x7f7c5806c0d0 msgr2=0x7f7c5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/2858970003 shutdown_connections 2024-03-16T20:06:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.331+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/2858970003 wait complete. 2024-03-16T20:06:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 Processor -- start 2024-03-16T20:06:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 -- start start 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c580724a0 0x7f7c580c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c58153fa0 0x7f7c580c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c581f3d00 con 0x7f7c58070840 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5815c240 con 0x7f7c58153fa0 2024-03-16T20:06:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c5d8f9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c581f3fc0 con 0x7f7c580724a0 2024-03-16T20:06:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c56ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c567fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c580724a0 0x7f7c580c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.335+0000 7f7c56ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49972/0 (socket says 172.21.15.32:49972) 2024-03-16T20:06:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.339+0000 7f7c56ffd640 1 -- 172.21.15.32:0/594931733 learned_addr learned my addr 172.21.15.32:0/594931733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.339+0000 7f7c577fe640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c58153fa0 0x7f7c580c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:25.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.343+0000 7f7c56ffd640 1 -- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c580724a0 msgr2=0x7f7c580c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.343+0000 7f7c56ffd640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c580724a0 0x7f7c580c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.343+0000 7f7c56ffd640 1 -- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c58153fa0 msgr2=0x7f7c580c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.343+0000 7f7c56ffd640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c58153fa0 0x7f7c580c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.343+0000 7f7c56ffd640 1 -- 172.21.15.32:0/594931733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0094d0 con 0x7f7c58070840 2024-03-16T20:06:25.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.347+0000 7f7c56ffd640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c3c0 tx=0x7f7c4800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c480052d0 con 0x7f7c58070840 2024-03-16T20:06:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c48005430 con 0x7f7c58070840 2024-03-16T20:06:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5815c500 con 0x7f7c58070840 2024-03-16T20:06:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c48003840 con 0x7f7c58070840 2024-03-16T20:06:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5815afb0 con 0x7f7c58070840 2024-03-16T20:06:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.351+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c58068000 con 0x7f7c58070840 2024-03-16T20:06:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.387+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7c480039a0 con 0x7f7c58070840 2024-03-16T20:06:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.391+0000 7f7c3ffff640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 0x7f7c44069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.391+0000 7f7c567fc640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 0x7f7c44069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.391+0000 7f7c567fc640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 0x7f7c44069350 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0b2000 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:25.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.395+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7c48054b90 con 0x7f7c58070840 2024-03-16T20:06:25.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.699+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7c48050e30 con 0x7f7c58070840 2024-03-16T20:06:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.931+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c58070c90 con 0x7f7c58070840 2024-03-16T20:06:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.951+0000 7f7c3ffff640 1 -- 172.21.15.32:0/594931733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f7c58070c90 con 0x7f7c58070840 2024-03-16T20:06:25.939 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:25.940 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 msgr2=0x7f7c44069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 0x7f7c44069350 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0b2000 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 msgr2=0x7f7c581f3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c3c0 tx=0x7f7c4800c880 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 shutdown_connections 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7c44066e90 0x7f7c44069350 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c58153fa0 0x7f7c580c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c580724a0 0x7f7c580c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 --2- 172.21.15.32:0/594931733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c58070840 0x7f7c581f3430 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 >> 172.21.15.32:0/594931733 conn(0x7f7c5806c0d0 msgr2=0x7f7c5806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 shutdown_connections 2024-03-16T20:06:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:25.955+0000 7f7c5d8f9640 1 -- 172.21.15.32:0/594931733 wait complete. 2024-03-16T20:06:26.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:26.571 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:27.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:27.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 Processor -- start 2024-03-16T20:06:27.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 -- start start 2024-03-16T20:06:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 0x7f49b8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49b815a420 0x7f49b815a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b8151020 con 0x7f49b80725e0 2024-03-16T20:06:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b815a000 con 0x7f49b815a420 2024-03-16T20:06:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.695+0000 7f49bda11640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b815add0 con 0x7f49b8150c40 2024-03-16T20:06:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.699+0000 7f49b6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 0x7f49b8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.699+0000 7f49b6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 0x7f49b8073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:36854/0 (socket says 172.21.15.32:36854) 2024-03-16T20:06:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.699+0000 7f49b6ffd640 1 -- 172.21.15.32:0/3805549797 learned_addr learned my addr 172.21.15.32:0/3805549797 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:27.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.699+0000 7f49b77fe640 1 --2- 172.21.15.32:0/3805549797 >> v2:172.21.15.32:3300/0 conn(0x7f49b815a420 0x7f49b815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.703+0000 7f49b77fe640 1 -- 172.21.15.32:0/3805549797 >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 msgr2=0x7f49b8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.703+0000 7f49b77fe640 1 --2- 172.21.15.32:0/3805549797 >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 0x7f49b8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.703+0000 7f49b77fe640 1 -- 172.21.15.32:0/3805549797 >> v1:172.21.15.59:6789/0 conn(0x7f49b80725e0 legacy=0x7f49b8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.703+0000 7f49b77fe640 1 -- 172.21.15.32:0/3805549797 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b8072da0 con 0x7f49b815a420 2024-03-16T20:06:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.707+0000 7f49b77fe640 1 --2- 172.21.15.32:0/3805549797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b815a420 0x7f49b815a890 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f49b806cf60 tx=0x7f49b001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab8b63b3ea8908b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49b67fc640 1 -- 172.21.15.32:0/3805549797 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b00a59f0 con 0x7f49b815a420 2024-03-16T20:06:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49b67fc640 1 -- 172.21.15.32:0/3805549797 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b00a5b50 con 0x7f49b815a420 2024-03-16T20:06:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49b67fc640 1 -- 172.21.15.32:0/3805549797 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b00ad440 con 0x7f49b815a420 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 -- 172.21.15.32:0/3805549797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b815a420 msgr2=0x7f49b815a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 --2- 172.21.15.32:0/3805549797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b815a420 0x7f49b815a890 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f49b806cf60 tx=0x7f49b001c930 comp rx=0 tx=0).stop 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 -- 172.21.15.32:0/3805549797 shutdown_connections 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 --2- 172.21.15.32:0/3805549797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b815a420 0x7f49b815a890 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 --2- 172.21.15.32:0/3805549797 >> v2:172.21.15.59:3300/0 conn(0x7f49b8150c40 0x7f49b8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 -- 172.21.15.32:0/3805549797 >> 172.21.15.32:0/3805549797 conn(0x7f49b806c0d0 msgr2=0x7f49b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 -- 172.21.15.32:0/3805549797 shutdown_connections 2024-03-16T20:06:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.715+0000 7f49bda11640 1 -- 172.21.15.32:0/3805549797 wait complete. 2024-03-16T20:06:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 Processor -- start 2024-03-16T20:06:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 -- start start 2024-03-16T20:06:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b80725e0 0x7f49b81f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49b815a420 0x7f49b814b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b8147c90 con 0x7f49b8150c40 2024-03-16T20:06:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b81484a0 con 0x7f49b80725e0 2024-03-16T20:06:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49bda11640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b814b320 con 0x7f49b815a420 2024-03-16T20:06:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.719+0000 7f49b77fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49b815a420 0x7f49b814b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.723+0000 7f49affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49994/0 (socket says 172.21.15.32:49994) 2024-03-16T20:06:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.723+0000 7f49affff640 1 -- 172.21.15.32:0/862736827 learned_addr learned my addr 172.21.15.32:0/862736827 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49b6ffd640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b80725e0 0x7f49b81f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49affff640 1 -- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49b815a420 msgr2=0x7f49b814b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:27.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49affff640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49b815a420 0x7f49b814b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:27.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49affff640 1 -- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b80725e0 msgr2=0x7f49b81f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:27.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49affff640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b80725e0 0x7f49b81f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:27.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.727+0000 7f49affff640 1 -- 172.21.15.32:0/862736827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0009a70 con 0x7f49b8150c40 2024-03-16T20:06:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.731+0000 7f49affff640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c3f0 tx=0x7f49a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a00052d0 con 0x7f49b8150c40 2024-03-16T20:06:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b815d050 con 0x7f49b8150c40 2024-03-16T20:06:27.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b81481c0 con 0x7f49b8150c40 2024-03-16T20:06:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b8068000 con 0x7f49b8150c40 2024-03-16T20:06:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a0001130 con 0x7f49b8150c40 2024-03-16T20:06:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:27.735+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0003600 con 0x7f49b8150c40 2024-03-16T20:06:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.015+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f49a0003760 con 0x7f49b8150c40 2024-03-16T20:06:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.015+0000 7f49b4ff9640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 0x7f4998069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.015+0000 7f49b6ffd640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 0x7f4998069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.015+0000 7f49b6ffd640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 0x7f4998069330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009980 tx=0x7f49a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:28.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.027+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f49a0054fc0 con 0x7f49b8150c40 2024-03-16T20:06:28.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.027+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f49a0054d90 con 0x7f49b8150c40 2024-03-16T20:06:28.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.219+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49b814b460 con 0x7f49b8150c40 2024-03-16T20:06:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.231+0000 7f49b4ff9640 1 -- 172.21.15.32:0/862736827 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f49a0017090 con 0x7f49b8150c40 2024-03-16T20:06:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:28.219 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 msgr2=0x7f4998069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 0x7f4998069330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009980 tx=0x7f49a8008040 comp rx=0 tx=0).stop 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 msgr2=0x7f49b814d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c3f0 tx=0x7f49a000c8b0 comp rx=0 tx=0).stop 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 shutdown_connections 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4998066e70 0x7f4998069330 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f49b815a420 0x7f49b814b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b8150c40 0x7f49b814d6d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 --2- 172.21.15.32:0/862736827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b80725e0 0x7f49b81f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 >> 172.21.15.32:0/862736827 conn(0x7f49b806c0d0 msgr2=0x7f49b814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 shutdown_connections 2024-03-16T20:06:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:28.235+0000 7f49bda11640 1 -- 172.21.15.32:0/862736827 wait complete. 2024-03-16T20:06:28.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:28.848 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:29.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.971+0000 7f85a441b640 1 Processor -- start 2024-03-16T20:06:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.971+0000 7f85a441b640 1 -- start start 2024-03-16T20:06:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a441b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f859c151ef0 0x7f859c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a441b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f859c15b2a0 0x7f859c15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a441b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c158c80 con 0x7f859c0c97b0 2024-03-16T20:06:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a441b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c0b0ad0 con 0x7f859c15b2a0 2024-03-16T20:06:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a441b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c157cb0 con 0x7f859c151ef0 2024-03-16T20:06:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a29b8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f859c15b2a0 0x7f859c15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a29b8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f859c15b2a0 0x7f859c15b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41902/0 (socket says 172.21.15.32:41902) 2024-03-16T20:06:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a29b8640 1 -- 172.21.15.32:0/2341116522 learned_addr learned my addr 172.21.15.32:0/2341116522 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.975+0000 7f85a21b7640 1 --2- 172.21.15.32:0/2341116522 >> v2:172.21.15.59:3300/0 conn(0x7f859c151ef0 0x7f859c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.979+0000 7f85a29b8640 1 -- 172.21.15.32:0/2341116522 >> v2:172.21.15.59:3300/0 conn(0x7f859c151ef0 msgr2=0x7f859c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.979+0000 7f85a29b8640 1 --2- 172.21.15.32:0/2341116522 >> v2:172.21.15.59:3300/0 conn(0x7f859c151ef0 0x7f859c0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.979+0000 7f85a29b8640 1 -- 172.21.15.32:0/2341116522 >> v1:172.21.15.32:6789/0 conn(0x7f859c0c97b0 legacy=0x7f859c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.979+0000 7f85a29b8640 1 -- 172.21.15.32:0/2341116522 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c15bfc0 con 0x7f859c15b2a0 2024-03-16T20:06:29.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.983+0000 7f85a29b8640 1 --2- 172.21.15.32:0/2341116522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c15b2a0 0x7f859c15b6e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f859c0cb3a0 tx=0x7f859401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e5c08e87f4f6ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a11b5640 1 -- 172.21.15.32:0/2341116522 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940af400 con 0x7f859c15b2a0 2024-03-16T20:06:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a11b5640 1 -- 172.21.15.32:0/2341116522 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85940ab050 con 0x7f859c15b2a0 2024-03-16T20:06:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a11b5640 1 -- 172.21.15.32:0/2341116522 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940af780 con 0x7f859c15b2a0 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 -- 172.21.15.32:0/2341116522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c15b2a0 msgr2=0x7f859c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 --2- 172.21.15.32:0/2341116522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c15b2a0 0x7f859c15b6e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f859c0cb3a0 tx=0x7f859401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 -- 172.21.15.32:0/2341116522 shutdown_connections 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 --2- 172.21.15.32:0/2341116522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c15b2a0 0x7f859c15b6e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 --2- 172.21.15.32:0/2341116522 >> v2:172.21.15.59:3300/0 conn(0x7f859c151ef0 0x7f859c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 -- 172.21.15.32:0/2341116522 >> 172.21.15.32:0/2341116522 conn(0x7f859c070f90 msgr2=0x7f859c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 -- 172.21.15.32:0/2341116522 shutdown_connections 2024-03-16T20:06:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.987+0000 7f85a441b640 1 -- 172.21.15.32:0/2341116522 wait complete. 2024-03-16T20:06:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 Processor -- start 2024-03-16T20:06:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 -- start start 2024-03-16T20:06:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f859c0c97b0 0x7f859c1f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f859c15b2a0 0x7f859c142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c06c610 con 0x7f859c151ef0 2024-03-16T20:06:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c06dd20 con 0x7f859c0c97b0 2024-03-16T20:06:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a441b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c06e000 con 0x7f859c15b2a0 2024-03-16T20:06:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a19b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.991+0000 7f85a19b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41910/0 (socket says 172.21.15.32:41910) 2024-03-16T20:06:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 -- 172.21.15.32:0/3415072492 learned_addr learned my addr 172.21.15.32:0/3415072492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a21b7640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f859c0c97b0 0x7f859c1f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a29b8640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f859c15b2a0 0x7f859c142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:29.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 -- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f859c15b2a0 msgr2=0x7f859c142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:29.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f859c15b2a0 0x7f859c142460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 -- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f859c0c97b0 msgr2=0x7f859c1f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:29.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f859c0c97b0 0x7f859c1f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:29.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.995+0000 7f85a19b6640 1 -- 172.21.15.32:0/3415072492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594009b30 con 0x7f859c151ef0 2024-03-16T20:06:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:29.999+0000 7f85a19b6640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f858c00c4d0 tx=0x7f858c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0052d0 con 0x7f859c151ef0 2024-03-16T20:06:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c005430 con 0x7f859c151ef0 2024-03-16T20:06:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c012830 con 0x7f859c151ef0 2024-03-16T20:06:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c0cb5e0 con 0x7f859c151ef0 2024-03-16T20:06:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c0cbba0 con 0x7f859c151ef0 2024-03-16T20:06:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.003+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c068000 con 0x7f859c151ef0 2024-03-16T20:06:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.027+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f858c012990 con 0x7f859c151ef0 2024-03-16T20:06:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.027+0000 7f8592ffd640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 0x7f8580069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.027+0000 7f85a21b7640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 0x7f8580069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.027+0000 7f85a21b7640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 0x7f8580069300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8598009b60 tx=0x7f8598009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.291+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f858c055b20 con 0x7f859c151ef0 2024-03-16T20:06:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.291+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f858c051dc0 con 0x7f859c151ef0 2024-03-16T20:06:30.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.507+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f859c0c8520 con 0x7f859c151ef0 2024-03-16T20:06:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.519+0000 7f8592ffd640 1 -- 172.21.15.32:0/3415072492 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f858c055480 con 0x7f859c151ef0 2024-03-16T20:06:30.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:30.507 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 msgr2=0x7f8580069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 0x7f8580069300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8598009b60 tx=0x7f8598009190 comp rx=0 tx=0).stop 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 msgr2=0x7f859c06c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f858c00c4d0 tx=0x7f858c00c990 comp rx=0 tx=0).stop 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 shutdown_connections 2024-03-16T20:06:30.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8580066e40 0x7f8580069300 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f859c15b2a0 0x7f859c142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859c151ef0 0x7f859c06c0d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 --2- 172.21.15.32:0/3415072492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f859c0c97b0 0x7f859c1f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 >> 172.21.15.32:0/3415072492 conn(0x7f859c070f90 msgr2=0x7f859c1443b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 shutdown_connections 2024-03-16T20:06:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:30.523+0000 7f85a441b640 1 -- 172.21.15.32:0/3415072492 wait complete. 2024-03-16T20:06:31.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:31.130 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:32.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:32.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 Processor -- start 2024-03-16T20:06:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 -- start start 2024-03-16T20:06:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b140c6040 0x7f3b140c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b14070760 con 0x7f3b140c6420 2024-03-16T20:06:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140731c0 con 0x7f3b14158390 2024-03-16T20:06:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.259+0000 7f3b19d0c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140734a0 con 0x7f3b140c6040 2024-03-16T20:06:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b140c6040 0x7f3b140c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b140c6040 0x7f3b140c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41912/0 (socket says 172.21.15.32:41912) 2024-03-16T20:06:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 -- 172.21.15.32:0/845636838 learned_addr learned my addr 172.21.15.32:0/845636838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 -- 172.21.15.32:0/845636838 >> v1:172.21.15.59:6789/0 conn(0x7f3b14158390 legacy=0x7f3b141516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 -- 172.21.15.32:0/845636838 >> v1:172.21.15.32:6789/0 conn(0x7f3b140c6420 legacy=0x7f3b14157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:32.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.263+0000 7f3b137fe640 1 -- 172.21.15.32:0/845636838 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1406fe30 con 0x7f3b140c6040 2024-03-16T20:06:32.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.267+0000 7f3b137fe640 1 --2- 172.21.15.32:0/845636838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b140c8800 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009870 tx=0x7f3b0401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf5f591fbc58138 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b127fc640 1 -- 172.21.15.32:0/845636838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040aa650 con 0x7f3b140c6040 2024-03-16T20:06:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b127fc640 1 -- 172.21.15.32:0/845636838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b040aa7b0 con 0x7f3b140c6040 2024-03-16T20:06:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b127fc640 1 -- 172.21.15.32:0/845636838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040aab40 con 0x7f3b140c6040 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- 172.21.15.32:0/845636838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 msgr2=0x7f3b140c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/845636838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b140c8800 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009870 tx=0x7f3b0401c4f0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- 172.21.15.32:0/845636838 shutdown_connections 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/845636838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b140c8800 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- 172.21.15.32:0/845636838 >> 172.21.15.32:0/845636838 conn(0x7f3b1406c0d0 msgr2=0x7f3b1406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- 172.21.15.32:0/845636838 shutdown_connections 2024-03-16T20:06:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- 172.21.15.32:0/845636838 wait complete. 2024-03-16T20:06:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 Processor -- start 2024-03-16T20:06:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.275+0000 7f3b19d0c640 1 -- start start 2024-03-16T20:06:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b140c6420 0x7f3b14157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b14158390 0x7f3b141576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1414e620 con 0x7f3b140c6040 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1414f360 con 0x7f3b14158390 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b19d0c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1414eda0 con 0x7f3b140c6420 2024-03-16T20:06:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b137fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b137fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41916/0 (socket says 172.21.15.32:41916) 2024-03-16T20:06:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b137fe640 1 -- 172.21.15.32:0/1366940395 learned_addr learned my addr 172.21.15.32:0/1366940395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.279+0000 7f3b13fff640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b14158390 0x7f3b141576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:32.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.283+0000 7f3b137fe640 1 -- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b140c6420 msgr2=0x7f3b14157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.283+0000 7f3b137fe640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b140c6420 0x7f3b14157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.283+0000 7f3b137fe640 1 -- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b14158390 msgr2=0x7f3b141576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.283+0000 7f3b137fe640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b14158390 0x7f3b141576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.283+0000 7f3b137fe640 1 -- 172.21.15.32:0/1366940395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b040094d0 con 0x7f3b140c6040 2024-03-16T20:06:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.287+0000 7f3b137fe640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3b040ac040 tx=0x7f3b0401ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.291+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040b38c0 con 0x7f3b140c6040 2024-03-16T20:06:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.291+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b040b3a20 con 0x7f3b140c6040 2024-03-16T20:06:32.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.295+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040b3dd0 con 0x7f3b140c6040 2024-03-16T20:06:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.295+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1414f080 con 0x7f3b140c6040 2024-03-16T20:06:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.295+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b140c54a0 con 0x7f3b140c6040 2024-03-16T20:06:32.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.295+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14068000 con 0x7f3b140c6040 2024-03-16T20:06:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.331+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3b040c6390 con 0x7f3b140c6040 2024-03-16T20:06:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.331+0000 7f3afbfff640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 0x7f3b00069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:32.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.331+0000 7f3b12ffd640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 0x7f3b00069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:32.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.331+0000 7f3b12ffd640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 0x7f3b00069350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b14156a60 tx=0x7f3b080079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:32.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.335+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3b040f0250 con 0x7f3b140c6040 2024-03-16T20:06:32.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.631+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3b040ec4f0 con 0x7f3b140c6040 2024-03-16T20:06:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.847+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b140c7d40 con 0x7f3b140c6040 2024-03-16T20:06:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.939+0000 7f3afbfff640 1 -- 172.21.15.32:0/1366940395 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f3b040efbb0 con 0x7f3b140c6040 2024-03-16T20:06:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:32.925 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 msgr2=0x7f3b00069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 0x7f3b00069350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b14156a60 tx=0x7f3b080079c0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 msgr2=0x7f3b141479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3b040ac040 tx=0x7f3b0401ad30 comp rx=0 tx=0).stop 2024-03-16T20:06:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 shutdown_connections 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b00066e90 0x7f3b00069350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b14158390 0x7f3b141576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b140c6420 0x7f3b14157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 --2- 172.21.15.32:0/1366940395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b140c6040 0x7f3b141479b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 >> 172.21.15.32:0/1366940395 conn(0x7f3b1406c0d0 msgr2=0x7f3b140cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 shutdown_connections 2024-03-16T20:06:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:32.943+0000 7f3b19d0c640 1 -- 172.21.15.32:0/1366940395 wait complete. 2024-03-16T20:06:33.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:33.551 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:34.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 Processor -- start 2024-03-16T20:06:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 -- start start 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08340725e0 0x7f0834070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08340708a0 0x7f083415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083415a000 con 0x7f08340725e0 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0834151020 con 0x7f08340708a0 2024-03-16T20:06:34.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0838978640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0834072da0 con 0x7f0834150c40 2024-03-16T20:06:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0831d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08340725e0 0x7f0834070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0831d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08340725e0 0x7f0834070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41918/0 (socket says 172.21.15.32:41918) 2024-03-16T20:06:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f0831d9b640 1 -- 172.21.15.32:0/2353335481 learned_addr learned my addr 172.21.15.32:0/2353335481 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.675+0000 7f083259c640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0832d9d640 1 --2- 172.21.15.32:0/2353335481 >> v2:172.21.15.59:3300/0 conn(0x7f08340708a0 0x7f083415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0831d9b640 1 -- 172.21.15.32:0/2353335481 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 msgr2=0x7f0834073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:34.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0831d9b640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0831d9b640 1 -- 172.21.15.32:0/2353335481 >> v2:172.21.15.59:3300/0 conn(0x7f08340708a0 msgr2=0x7f083415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:34.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0831d9b640 1 --2- 172.21.15.32:0/2353335481 >> v2:172.21.15.59:3300/0 conn(0x7f08340708a0 0x7f083415a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.679+0000 7f0831d9b640 1 -- 172.21.15.32:0/2353335481 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083415e310 con 0x7f08340725e0 2024-03-16T20:06:34.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.683+0000 7f0831d9b640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f0834070360 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f083415e500 tx=0x7f082801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bade3ae9bd7774 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.687+0000 7f083159a640 1 -- 172.21.15.32:0/2353335481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280ab5f0 con 0x7f08340725e0 2024-03-16T20:06:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.687+0000 7f083159a640 1 -- 172.21.15.32:0/2353335481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08280ab750 con 0x7f08340725e0 2024-03-16T20:06:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.687+0000 7f083159a640 1 -- 172.21.15.32:0/2353335481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280abae0 con 0x7f08340725e0 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- 172.21.15.32:0/2353335481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 msgr2=0x7f0834070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f0834070360 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f083415e500 tx=0x7f082801c520 comp rx=0 tx=0).stop 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- 172.21.15.32:0/2353335481 shutdown_connections 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- 172.21.15.32:0/2353335481 >> v2:172.21.15.59:3300/0 conn(0x7f08340708a0 0x7f083415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f0834070360 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- 172.21.15.32:0/2353335481 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- 172.21.15.32:0/2353335481 >> 172.21.15.32:0/2353335481 conn(0x7f083406c0d0 msgr2=0x7f083406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- 172.21.15.32:0/2353335481 shutdown_connections 2024-03-16T20:06:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- 172.21.15.32:0/2353335481 wait complete. 2024-03-16T20:06:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 Processor -- start 2024-03-16T20:06:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- start start 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08340708a0 0x7f08340c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08341f6f90 con 0x7f08340725e0 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340c6610 con 0x7f08340708a0 2024-03-16T20:06:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.691+0000 7f0838978640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08340c6090 con 0x7f0834150c40 2024-03-16T20:06:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.695+0000 7f0831d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.695+0000 7f0831d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41926/0 (socket says 172.21.15.32:41926) 2024-03-16T20:06:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.695+0000 7f0831d9b640 1 -- 172.21.15.32:0/90945837 learned_addr learned my addr 172.21.15.32:0/90945837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.695+0000 7f0832d9d640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.695+0000 7f083259c640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08340708a0 0x7f08340c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.699+0000 7f0831d9b640 1 -- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 msgr2=0x7f0834146610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.699+0000 7f0831d9b640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834146610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.699+0000 7f0831d9b640 1 -- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08340708a0 msgr2=0x7f08340c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.699+0000 7f0831d9b640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08340708a0 0x7f08340c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.699+0000 7f0831d9b640 1 -- 172.21.15.32:0/90945837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280094d0 con 0x7f08340725e0 2024-03-16T20:06:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.703+0000 7f0831d9b640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f08280ad040 tx=0x7f082801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:34.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.707+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280be5f0 con 0x7f08340725e0 2024-03-16T20:06:34.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.707+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08280be750 con 0x7f08340725e0 2024-03-16T20:06:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.707+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08340c3ac0 con 0x7f08340725e0 2024-03-16T20:06:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.707+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280beb00 con 0x7f08340725e0 2024-03-16T20:06:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.711+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08340c6f30 con 0x7f08340725e0 2024-03-16T20:06:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.711+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08340c6d90 con 0x7f08340725e0 2024-03-16T20:06:34.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.751+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08280b43f0 con 0x7f08340725e0 2024-03-16T20:06:34.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.751+0000 7f0822ffd640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 0x7f0814069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:34.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.751+0000 7f083259c640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 0x7f0814069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:34.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.751+0000 7f083259c640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 0x7f0814069300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0824009ac0 tx=0x7f0824009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:34.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:34.755+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f08280a8050 con 0x7f08340725e0 2024-03-16T20:06:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.031+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f08280ec5a0 con 0x7f08340725e0 2024-03-16T20:06:35.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.299+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08340c6d90 con 0x7f08340725e0 2024-03-16T20:06:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.315+0000 7f0822ffd640 1 -- 172.21.15.32:0/90945837 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f08340c6d90 con 0x7f08340725e0 2024-03-16T20:06:35.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:35.304 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 msgr2=0x7f0814069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 0x7f0814069300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0824009ac0 tx=0x7f0824009190 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 msgr2=0x7f08341460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f08280ad040 tx=0x7f082801ad60 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 shutdown_connections 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0814066e40 0x7f0814069300 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834150c40 0x7f0834146610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08340725e0 0x7f08341460d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 --2- 172.21.15.32:0/90945837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08340708a0 0x7f08340c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 >> 172.21.15.32:0/90945837 conn(0x7f083406c0d0 msgr2=0x7f08340cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 shutdown_connections 2024-03-16T20:06:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:35.319+0000 7f0838978640 1 -- 172.21.15.32:0/90945837 wait complete. 2024-03-16T20:06:35.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:35.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:36.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:37.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.055+0000 7f6b3979b640 1 Processor -- start 2024-03-16T20:06:37.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.055+0000 7f6b3979b640 1 -- start start 2024-03-16T20:06:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b3979b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b340c81e0 0x7f6b340c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b3979b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b3415b2a0 0x7f6b3415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b3979b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b34158c80 con 0x7f6b34151ef0 2024-03-16T20:06:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b3979b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340b0ad0 con 0x7f6b340c81e0 2024-03-16T20:06:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b3979b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34157cb0 con 0x7f6b3415b2a0 2024-03-16T20:06:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b327fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b340c81e0 0x7f6b340c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b327fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b340c81e0 0x7f6b340c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41940/0 (socket says 172.21.15.32:41940) 2024-03-16T20:06:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b327fc640 1 -- 172.21.15.32:0/2894009290 learned_addr learned my addr 172.21.15.32:0/2894009290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.059+0000 7f6b337fe640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b3415b2a0 0x7f6b3415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.063+0000 7f6b327fc640 1 -- 172.21.15.32:0/2894009290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b3415b2a0 msgr2=0x7f6b3415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.063+0000 7f6b327fc640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b3415b2a0 0x7f6b3415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.063+0000 7f6b327fc640 1 -- 172.21.15.32:0/2894009290 >> v1:172.21.15.32:6789/0 conn(0x7f6b34151ef0 legacy=0x7f6b340c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.063+0000 7f6b327fc640 1 -- 172.21.15.32:0/2894009290 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3415bfc0 con 0x7f6b340c81e0 2024-03-16T20:06:37.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.067+0000 7f6b327fc640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b340c81e0 0x7f6b340c8630 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009870 tx=0x7f6b2801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8955a52606468766 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:37.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.071+0000 7f6b31ffb640 1 -- 172.21.15.32:0/2894009290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280a89d0 con 0x7f6b340c81e0 2024-03-16T20:06:37.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.071+0000 7f6b31ffb640 1 -- 172.21.15.32:0/2894009290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b280a8b30 con 0x7f6b340c81e0 2024-03-16T20:06:37.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.071+0000 7f6b31ffb640 1 -- 172.21.15.32:0/2894009290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280b0440 con 0x7f6b340c81e0 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- 172.21.15.32:0/2894009290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b340c81e0 msgr2=0x7f6b340c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b340c81e0 0x7f6b340c8630 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009870 tx=0x7f6b2801ed10 comp rx=0 tx=0).stop 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- 172.21.15.32:0/2894009290 shutdown_connections 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b3415b2a0 0x7f6b3415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- 172.21.15.32:0/2894009290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b340c81e0 0x7f6b340c8630 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- 172.21.15.32:0/2894009290 >> 172.21.15.32:0/2894009290 conn(0x7f6b34070f90 msgr2=0x7f6b340713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- 172.21.15.32:0/2894009290 shutdown_connections 2024-03-16T20:06:37.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- 172.21.15.32:0/2894009290 wait complete. 2024-03-16T20:06:37.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 Processor -- start 2024-03-16T20:06:37.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- start start 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b340c81e0 0x7f6b34149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b3415b2a0 0x7f6b341f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34155d40 con 0x7f6b34151ef0 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340b1690 con 0x7f6b3415b2a0 2024-03-16T20:06:37.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.075+0000 7f6b3979b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340b1930 con 0x7f6b340c81e0 2024-03-16T20:06:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.079+0000 7f6b327fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.079+0000 7f6b32ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b340c81e0 0x7f6b34149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.079+0000 7f6b327fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41948/0 (socket says 172.21.15.32:41948) 2024-03-16T20:06:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.079+0000 7f6b327fc640 1 -- 172.21.15.32:0/1442657011 learned_addr learned my addr 172.21.15.32:0/1442657011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.079+0000 7f6b337fe640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b3415b2a0 0x7f6b341f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.083+0000 7f6b327fc640 1 -- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b340c81e0 msgr2=0x7f6b34149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:37.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.083+0000 7f6b327fc640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b340c81e0 0x7f6b34149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.083+0000 7f6b327fc640 1 -- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b3415b2a0 msgr2=0x7f6b341f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.083+0000 7f6b327fc640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b3415b2a0 0x7f6b341f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.083+0000 7f6b327fc640 1 -- 172.21.15.32:0/1442657011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280094d0 con 0x7f6b34151ef0 2024-03-16T20:06:37.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.087+0000 7f6b327fc640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6b280b3040 tx=0x7f6b2801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280b0e10 con 0x7f6b34151ef0 2024-03-16T20:06:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b280bb480 con 0x7f6b34151ef0 2024-03-16T20:06:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280bb830 con 0x7f6b34151ef0 2024-03-16T20:06:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b340b1c10 con 0x7f6b34151ef0 2024-03-16T20:06:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b341e1040 con 0x7f6b34151ef0 2024-03-16T20:06:37.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.091+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34068000 con 0x7f6b34151ef0 2024-03-16T20:06:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.127+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6b280bb990 con 0x7f6b34151ef0 2024-03-16T20:06:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.127+0000 7f6b1b7fe640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 0x7f6b20069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.131+0000 7f6b32ffd640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 0x7f6b20069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.131+0000 7f6b32ffd640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 0x7f6b20069350 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6b24007fd0 tx=0x7f6b24007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.135+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6b280f4560 con 0x7f6b34151ef0 2024-03-16T20:06:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.407+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6b280f0800 con 0x7f6b34151ef0 2024-03-16T20:06:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.719+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b34068000 con 0x7f6b34151ef0 2024-03-16T20:06:37.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b1b7fe640 1 -- 172.21.15.32:0/1442657011 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f6b280b5020 con 0x7f6b34151ef0 2024-03-16T20:06:37.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:37.752 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:37.753 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:37.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 msgr2=0x7f6b20069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 0x7f6b20069350 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6b24007fd0 tx=0x7f6b24007e40 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 msgr2=0x7f6b341e1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6b280b3040 tx=0x7f6b2801bab0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 shutdown_connections 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6b20066e90 0x7f6b20069350 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b3415b2a0 0x7f6b341f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b34151ef0 0x7f6b341e1b20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 --2- 172.21.15.32:0/1442657011 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b340c81e0 0x7f6b34149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 >> 172.21.15.32:0/1442657011 conn(0x7f6b34070f90 msgr2=0x7f6b3414b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 shutdown_connections 2024-03-16T20:06:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:37.767+0000 7f6b3979b640 1 -- 172.21.15.32:0/1442657011 wait complete. 2024-03-16T20:06:38.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:38.377 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:39.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:39.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 Processor -- start 2024-03-16T20:06:39.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 -- start start 2024-03-16T20:06:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 0x7fc3380ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3380c2910 0x7fc3380cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3380ca1d0 con 0x7fc338071110 2024-03-16T20:06:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3380c34a0 con 0x7fc3380c2910 2024-03-16T20:06:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.499+0000 7fc33e036640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380c9ba0 con 0x7fc3380714f0 2024-03-16T20:06:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc3377fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc338071110 0x7fc3380c82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58664/0 (socket says 172.21.15.32:58664) 2024-03-16T20:06:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc3377fe640 1 -- 172.21.15.32:0/2370725132 learned_addr learned my addr 172.21.15.32:0/2370725132 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc337fff640 1 --2- 172.21.15.32:0/2370725132 >> v2:172.21.15.59:3300/0 conn(0x7fc3380c2910 0x7fc3380cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc336ffd640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 0x7fc3380ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924489436 0 0) 0x7fc3380ca1d0 con 0x7fc338071110 2024-03-16T20:06:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.503+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc320003530 con 0x7fc338071110 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674458711 0 0) 0x7fc320003530 con 0x7fc338071110 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 msgr2=0x7fc3380ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 0x7fc3380ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 >> v2:172.21.15.59:3300/0 conn(0x7fc3380c2910 msgr2=0x7fc3380cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 --2- 172.21.15.32:0/2370725132 >> v2:172.21.15.59:3300/0 conn(0x7fc3380c2910 0x7fc3380cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380c2ee0 con 0x7fc338071110 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fc328001680 con 0x7fc338071110 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.507+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 >> v1:172.21.15.32:6789/0 conn(0x7fc338071110 legacy=0x7fc3380c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:39.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc320003980 0x7fc320003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3200047d0 0x7fc320004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 0x7fc320005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380c34a0 con 0x7fc3200047d0 2024-03-16T20:06:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380c9ba0 con 0x7fc320003980 2024-03-16T20:06:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32000b9c0 con 0x7fc3380714f0 2024-03-16T20:06:39.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.511+0000 7fc336ffd640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3200047d0 0x7fc320004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.515+0000 7fc336ffd640 1 -- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 msgr2=0x7fc320005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.515+0000 7fc336ffd640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc3380714f0 0x7fc320005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.515+0000 7fc336ffd640 1 -- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc320003980 msgr2=0x7fc320003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.515+0000 7fc336ffd640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc320003980 0x7fc320003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.515+0000 7fc336ffd640 1 -- 172.21.15.32:0/2370725132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc320003530 con 0x7fc3200047d0 2024-03-16T20:06:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.519+0000 7fc336ffd640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3200047d0 0x7fc320004c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009a90 tx=0x7fc32c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79392bc34bed759a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c0a5580 con 0x7fc3200047d0 2024-03-16T20:06:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c0a56e0 con 0x7fc3200047d0 2024-03-16T20:06:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc3367fc640 1 -- 172.21.15.32:0/2370725132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c0a5a70 con 0x7fc3200047d0 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 -- 172.21.15.32:0/2370725132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3200047d0 msgr2=0x7fc320004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 --2- 172.21.15.32:0/2370725132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3200047d0 0x7fc320004c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009a90 tx=0x7fc32c01ca70 comp rx=0 tx=0).stop 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc337fff640 1 -- 172.21.15.32:0/2370725132 reap_dead start 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 -- 172.21.15.32:0/2370725132 shutdown_connections 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 -- 172.21.15.32:0/2370725132 >> 172.21.15.32:0/2370725132 conn(0x7fc33806afc0 msgr2=0x7fc33806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 -- 172.21.15.32:0/2370725132 shutdown_connections 2024-03-16T20:06:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.523+0000 7fc33e036640 1 -- 172.21.15.32:0/2370725132 wait complete. 2024-03-16T20:06:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 Processor -- start 2024-03-16T20:06:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 -- start start 2024-03-16T20:06:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc33814a530 0x7fc338154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc338155420 0x7fc338158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338156cf0 con 0x7fc33814a910 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381eb830 con 0x7fc338155420 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc33e036640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc338155bf0 con 0x7fc33814a530 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc337fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc337fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51274/0 (socket says 172.21.15.32:51274) 2024-03-16T20:06:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc337fff640 1 -- 172.21.15.32:0/1861101343 learned_addr learned my addr 172.21.15.32:0/1861101343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:39.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.527+0000 7fc3377fe640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc33814a530 0x7fc338154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc336ffd640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc338155420 0x7fc338158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc337fff640 1 -- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc33814a530 msgr2=0x7fc338154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc337fff640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc33814a530 0x7fc338154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc337fff640 1 -- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc338155420 msgr2=0x7fc338158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc337fff640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc338155420 0x7fc338158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.531+0000 7fc337fff640 1 -- 172.21.15.32:0/1861101343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380b2e20 con 0x7fc33814a910 2024-03-16T20:06:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.535+0000 7fc337fff640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc338071c40 tx=0x7fc33000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330010040 con 0x7fc33814a910 2024-03-16T20:06:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33000bae0 con 0x7fc33814a910 2024-03-16T20:06:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc33e036640 1 -- 172.21.15.32:0/1861101343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc338157e50 con 0x7fc33814a910 2024-03-16T20:06:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33000be70 con 0x7fc33814a910 2024-03-16T20:06:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc33e036640 1 -- 172.21.15.32:0/1861101343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338156170 con 0x7fc33814a910 2024-03-16T20:06:39.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.539+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3100052a0 con 0x7fc33814a910 2024-03-16T20:06:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.571+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc330028430 con 0x7fc33814a910 2024-03-16T20:06:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.575+0000 7fc31ffff640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 0x7fc324069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.575+0000 7fc3377fe640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 0x7fc324069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:39.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.575+0000 7fc3377fe640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 0x7fc324069350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc328001060 tx=0x7fc3280019c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.843+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc330056a00 con 0x7fc33814a910 2024-03-16T20:06:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:39.843+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc3300560d0 con 0x7fc33814a910 2024-03-16T20:06:40.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.051+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc310005900 con 0x7fc33814a910 2024-03-16T20:06:40.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.075+0000 7fc31ffff640 1 -- 172.21.15.32:0/1861101343 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc310005900 con 0x7fc33814a910 2024-03-16T20:06:40.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.075+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 msgr2=0x7fc324069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.075+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 0x7fc324069350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc328001060 tx=0x7fc3280019c0 comp rx=0 tx=0).stop 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.075+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 msgr2=0x7fc3381ebcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.075+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc338071c40 tx=0x7fc33000b480 comp rx=0 tx=0).stop 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 shutdown_connections 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc324066e90 0x7fc324069350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33814a910 0x7fc3381ebcf0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc338155420 0x7fc338158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 --2- 172.21.15.32:0/1861101343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc33814a530 0x7fc338154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 >> 172.21.15.32:0/1861101343 conn(0x7fc33806afc0 msgr2=0x7fc33814db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 shutdown_connections 2024-03-16T20:06:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:40.079+0000 7fc31dffb640 1 -- 172.21.15.32:0/1861101343 wait complete. 2024-03-16T20:06:40.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:40.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:41.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:41.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 Processor -- start 2024-03-16T20:06:41.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 -- start start 2024-03-16T20:06:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0d280c8b90 0x7f0d2815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0d2815df30 0x7f0d280c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2815ad50 con 0x7f0d28153fc0 2024-03-16T20:06:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d280c7ce0 con 0x7f0d280c8b90 2024-03-16T20:06:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2eb43640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2815a1f0 con 0x7f0d2815df30 2024-03-16T20:06:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d27fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0d280c8b90 0x7f0d2815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d27fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0d280c8b90 0x7f0d2815d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51278/0 (socket says 172.21.15.32:51278) 2024-03-16T20:06:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d27fff640 1 -- 172.21.15.32:0/2608771409 learned_addr learned my addr 172.21.15.32:0/2608771409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.815+0000 7f0d2d0e0640 1 --2- 172.21.15.32:0/2608771409 >> v2:172.21.15.59:3300/0 conn(0x7f0d2815df30 0x7f0d280c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.819+0000 7f0d27fff640 1 -- 172.21.15.32:0/2608771409 >> v2:172.21.15.59:3300/0 conn(0x7f0d2815df30 msgr2=0x7f0d280c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.819+0000 7f0d27fff640 1 --2- 172.21.15.32:0/2608771409 >> v2:172.21.15.59:3300/0 conn(0x7f0d2815df30 0x7f0d280c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.819+0000 7f0d27fff640 1 -- 172.21.15.32:0/2608771409 >> v1:172.21.15.32:6789/0 conn(0x7f0d28153fc0 legacy=0x7f0d280c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.819+0000 7f0d27fff640 1 -- 172.21.15.32:0/2608771409 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2815a4b0 con 0x7f0d280c8b90 2024-03-16T20:06:41.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.823+0000 7f0d27fff640 1 --2- 172.21.15.32:0/2608771409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d2815d9f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0d280c4830 tx=0x7f0d1401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25656b06b515ed09 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:41.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d277fe640 1 -- 172.21.15.32:0/2608771409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140a89d0 con 0x7f0d280c8b90 2024-03-16T20:06:41.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d277fe640 1 -- 172.21.15.32:0/2608771409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d140a8b30 con 0x7f0d280c8b90 2024-03-16T20:06:41.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d277fe640 1 -- 172.21.15.32:0/2608771409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140b0440 con 0x7f0d280c8b90 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- 172.21.15.32:0/2608771409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 msgr2=0x7f0d2815d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/2608771409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d2815d9f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0d280c4830 tx=0x7f0d1401ecd0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- 172.21.15.32:0/2608771409 shutdown_connections 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/2608771409 >> v2:172.21.15.59:3300/0 conn(0x7f0d2815df30 0x7f0d280c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/2608771409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d2815d9f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- 172.21.15.32:0/2608771409 >> 172.21.15.32:0/2608771409 conn(0x7f0d280cc7a0 msgr2=0x7f0d280ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- 172.21.15.32:0/2608771409 shutdown_connections 2024-03-16T20:06:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- 172.21.15.32:0/2608771409 wait complete. 2024-03-16T20:06:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 Processor -- start 2024-03-16T20:06:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.831+0000 7f0d2eb43640 1 -- start start 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d28153fc0 0x7f0d2814c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d2815df30 0x7f0d281f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d281e47e0 con 0x7f0d280c8b90 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d281e3250 con 0x7f0d28153fc0 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2eb43640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d281f38e0 con 0x7f0d2815df30 2024-03-16T20:06:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2c8df640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2c8df640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51284/0 (socket says 172.21.15.32:51284) 2024-03-16T20:06:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2c8df640 1 -- 172.21.15.32:0/1543991917 learned_addr learned my addr 172.21.15.32:0/1543991917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d27fff640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d28153fc0 0x7f0d2814c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.835+0000 7f0d2d0e0640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d2815df30 0x7f0d281f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:41.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.839+0000 7f0d2c8df640 1 -- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d2815df30 msgr2=0x7f0d281f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:41.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.839+0000 7f0d2c8df640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d2815df30 0x7f0d281f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.839+0000 7f0d2c8df640 1 -- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d28153fc0 msgr2=0x7f0d2814c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:41.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.839+0000 7f0d2c8df640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d28153fc0 0x7f0d2814c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:41.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.839+0000 7f0d2c8df640 1 -- 172.21.15.32:0/1543991917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140094d0 con 0x7f0d280c8b90 2024-03-16T20:06:41.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.843+0000 7f0d2c8df640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00b410 tx=0x7f0d1c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:41.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.847+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c011830 con 0x7f0d280c8b90 2024-03-16T20:06:41.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.847+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c011990 con 0x7f0d280c8b90 2024-03-16T20:06:41.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.847+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d28152e60 con 0x7f0d280c8b90 2024-03-16T20:06:41.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.847+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c011d40 con 0x7f0d280c8b90 2024-03-16T20:06:41.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.847+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d281e3780 con 0x7f0d280c8b90 2024-03-16T20:06:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:41.851+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d28068000 con 0x7f0d280c8b90 2024-03-16T20:06:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.131+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d1c01a400 con 0x7f0d280c8b90 2024-03-16T20:06:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.131+0000 7f0d257fa640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 0x7f0d0c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.131+0000 7f0d27fff640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 0x7f0d0c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.131+0000 7f0d27fff640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 0x7f0d0c069350 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0d140b2000 tx=0x7f0d14009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.147+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0d1c055c00 con 0x7f0d280c8b90 2024-03-16T20:06:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.147+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0d1c051ea0 con 0x7f0d280c8b90 2024-03-16T20:06:42.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.467+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d280cd0a0 con 0x7f0d280c8b90 2024-03-16T20:06:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d257fa640 1 -- 172.21.15.32:0/1543991917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f0d1c015090 con 0x7f0d280c8b90 2024-03-16T20:06:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:42.481 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 msgr2=0x7f0d0c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 0x7f0d0c069350 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0d140b2000 tx=0x7f0d14009040 comp rx=0 tx=0).stop 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 msgr2=0x7f0d28151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00b410 tx=0x7f0d1c00b8d0 comp rx=0 tx=0).stop 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 shutdown_connections 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0d0c066e90 0x7f0d0c069350 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d2815df30 0x7f0d281f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d28153fc0 0x7f0d2814c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 --2- 172.21.15.32:0/1543991917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d280c8b90 0x7f0d28151a40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 >> 172.21.15.32:0/1543991917 conn(0x7f0d280cc7a0 msgr2=0x7f0d280c53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 shutdown_connections 2024-03-16T20:06:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:42.495+0000 7f0d2eb43640 1 -- 172.21.15.32:0/1543991917 wait complete. 2024-03-16T20:06:42.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:06:42.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:06:43.104 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:43.104 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:44.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:44.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.227+0000 7f108d70c640 1 Processor -- start 2024-03-16T20:06:44.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 -- start start 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 0x7f10880c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1088158390 0x7f10881516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1088070760 con 0x7f10880c6420 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10880731c0 con 0x7f1088158390 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f108d70c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880734a0 con 0x7f10880c6040 2024-03-16T20:06:44.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f10867fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f10880c6420 0x7f1088157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58688/0 (socket says 172.21.15.32:58688) 2024-03-16T20:06:44.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f10867fc640 1 -- 172.21.15.32:0/1021787886 learned_addr learned my addr 172.21.15.32:0/1021787886 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:44.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.231+0000 7f1086ffd640 1 --2- 172.21.15.32:0/1021787886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 0x7f10880c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1085ffb640 1 -- 172.21.15.32:0/1021787886 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162252977 0 0) 0x7f1088070760 con 0x7f10880c6420 2024-03-16T20:06:44.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1085ffb640 1 -- 172.21.15.32:0/1021787886 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1070003530 con 0x7f10880c6420 2024-03-16T20:06:44.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f10877fe640 1 --2- 172.21.15.32:0/1021787886 >> v2:172.21.15.59:3300/0 conn(0x7f1088158390 0x7f10881516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1086ffd640 1 -- 172.21.15.32:0/1021787886 >> v2:172.21.15.59:3300/0 conn(0x7f1088158390 msgr2=0x7f10881516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1086ffd640 1 --2- 172.21.15.32:0/1021787886 >> v2:172.21.15.59:3300/0 conn(0x7f1088158390 0x7f10881516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1086ffd640 1 -- 172.21.15.32:0/1021787886 >> v1:172.21.15.32:6789/0 conn(0x7f10880c6420 legacy=0x7f1088157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.235+0000 7f1086ffd640 1 -- 172.21.15.32:0/1021787886 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108806fe30 con 0x7f10880c6040 2024-03-16T20:06:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.239+0000 7f1086ffd640 1 --2- 172.21.15.32:0/1021787886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 0x7f10880c8800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f10780097f0 tx=0x7f107801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d43ecba37c84a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f1085ffb640 1 -- 172.21.15.32:0/1021787886 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780ab670 con 0x7f10880c6040 2024-03-16T20:06:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f1085ffb640 1 -- 172.21.15.32:0/1021787886 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10780ab7d0 con 0x7f10880c6040 2024-03-16T20:06:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f1085ffb640 1 -- 172.21.15.32:0/1021787886 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780abb60 con 0x7f10880c6040 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 -- 172.21.15.32:0/1021787886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 msgr2=0x7f10880c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 --2- 172.21.15.32:0/1021787886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 0x7f10880c8800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f10780097f0 tx=0x7f107801c470 comp rx=0 tx=0).stop 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 -- 172.21.15.32:0/1021787886 shutdown_connections 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 --2- 172.21.15.32:0/1021787886 >> v2:172.21.15.59:3300/0 conn(0x7f1088158390 0x7f10881516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 --2- 172.21.15.32:0/1021787886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6040 0x7f10880c8800 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 -- 172.21.15.32:0/1021787886 >> 172.21.15.32:0/1021787886 conn(0x7f108806c0d0 msgr2=0x7f108806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 -- 172.21.15.32:0/1021787886 shutdown_connections 2024-03-16T20:06:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.247+0000 7f108d70c640 1 -- 172.21.15.32:0/1021787886 wait complete. 2024-03-16T20:06:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 Processor -- start 2024-03-16T20:06:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 -- start start 2024-03-16T20:06:44.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6420 0x7f10881579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1088158390 0x7f10881f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10881555e0 con 0x7f10880c6040 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108815bcc0 con 0x7f1088158390 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f108d70c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10881f3550 con 0x7f10880c6420 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51296/0 (socket says 172.21.15.32:51296) 2024-03-16T20:06:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f1086ffd640 1 -- 172.21.15.32:0/4280241350 learned_addr learned my addr 172.21.15.32:0/4280241350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.251+0000 7f10877fe640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1088158390 0x7f10881f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.255+0000 7f10867fc640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6420 0x7f10881579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.259+0000 7f1086ffd640 1 -- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6420 msgr2=0x7f10881579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.259+0000 7f1086ffd640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6420 0x7f10881579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.259+0000 7f1086ffd640 1 -- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1088158390 msgr2=0x7f10881f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.259+0000 7f1086ffd640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1088158390 0x7f10881f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.259+0000 7f1086ffd640 1 -- 172.21.15.32:0/4280241350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009420 con 0x7f10880c6040 2024-03-16T20:06:44.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.263+0000 7f1086ffd640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f10780ad040 tx=0x7f107801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:44.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.287+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780b5810 con 0x7f10880c6040 2024-03-16T20:06:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.287+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10881f3830 con 0x7f10880c6040 2024-03-16T20:06:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.287+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10881f3df0 con 0x7f10880c6040 2024-03-16T20:06:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.291+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10881f6770 con 0x7f10880c6040 2024-03-16T20:06:44.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.291+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10780ab630 con 0x7f10880c6040 2024-03-16T20:06:44.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.291+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780b5b70 con 0x7f10880c6040 2024-03-16T20:06:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.563+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f10780b5cd0 con 0x7f10880c6040 2024-03-16T20:06:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.563+0000 7f106f7fe640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 0x7f1074069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.563+0000 7f10867fc640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 0x7f1074069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.563+0000 7f10867fc640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 0x7f1074069300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10881f7870 tx=0x7f107c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.607+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f10780f06b0 con 0x7f10880c6040 2024-03-16T20:06:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.607+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f10780ec950 con 0x7f10880c6040 2024-03-16T20:06:44.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.819+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10881f6770 con 0x7f10880c6040 2024-03-16T20:06:44.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.835+0000 7f106f7fe640 1 -- 172.21.15.32:0/4280241350 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f10881f6770 con 0x7f10880c6040 2024-03-16T20:06:44.823 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:44.823 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:44.823 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 msgr2=0x7f1074069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 0x7f1074069300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10881f7870 tx=0x7f107c009040 comp rx=0 tx=0).stop 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 msgr2=0x7f10881550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f10780ad040 tx=0x7f107801acb0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 shutdown_connections 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1074066e40 0x7f1074069300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1088158390 0x7f10881f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10880c6420 0x7f10881579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 --2- 172.21.15.32:0/4280241350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10880c6040 0x7f10881550a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 >> 172.21.15.32:0/4280241350 conn(0x7f108806c0d0 msgr2=0x7f108806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 shutdown_connections 2024-03-16T20:06:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:44.839+0000 7f108d70c640 1 -- 172.21.15.32:0/4280241350 wait complete. 2024-03-16T20:06:45.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:45.454 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:46.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:46.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.574+0000 7f30e797f640 1 Processor -- start 2024-03-16T20:06:46.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 -- start start 2024-03-16T20:06:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e00c81e0 0x7f30e00c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30e015b2a0 0x7f30e015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e0158c80 con 0x7f30e0151ef0 2024-03-16T20:06:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e00b0ad0 con 0x7f30e015b2a0 2024-03-16T20:06:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e797f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0157cb0 con 0x7f30e00c81e0 2024-03-16T20:06:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e571b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30e0151ef0 0x7f30e00c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58690/0 (socket says 172.21.15.32:58690) 2024-03-16T20:06:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e571b640 1 -- 172.21.15.32:0/964764350 learned_addr learned my addr 172.21.15.32:0/964764350 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e4f1a640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e00c81e0 0x7f30e00c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.578+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/964764350 >> v2:172.21.15.59:3300/0 conn(0x7f30e015b2a0 0x7f30e015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30d7fff640 1 -- 172.21.15.32:0/964764350 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303285871 0 0) 0x7f30e0158c80 con 0x7f30e0151ef0 2024-03-16T20:06:46.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30d7fff640 1 -- 172.21.15.32:0/964764350 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc003530 con 0x7f30e0151ef0 2024-03-16T20:06:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30e5f1c640 1 -- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e00c81e0 msgr2=0x7f30e00c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e00c81e0 0x7f30e00c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30e5f1c640 1 -- 172.21.15.32:0/964764350 >> v1:172.21.15.32:6789/0 conn(0x7f30e0151ef0 legacy=0x7f30e00c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.582+0000 7f30e5f1c640 1 -- 172.21.15.32:0/964764350 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e015bfc0 con 0x7f30e015b2a0 2024-03-16T20:06:46.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.590+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e015b2a0 0x7f30e015b6d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009d90 tx=0x7f30d801f170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba4ca49d127bbfe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:46.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.594+0000 7f30d7fff640 1 -- 172.21.15.32:0/964764350 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80af600 con 0x7f30e015b2a0 2024-03-16T20:06:46.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.594+0000 7f30d7fff640 1 -- 172.21.15.32:0/964764350 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d80ab050 con 0x7f30e015b2a0 2024-03-16T20:06:46.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.594+0000 7f30d7fff640 1 -- 172.21.15.32:0/964764350 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80af980 con 0x7f30e015b2a0 2024-03-16T20:06:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e015b2a0 msgr2=0x7f30e015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e015b2a0 0x7f30e015b6d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009d90 tx=0x7f30d801f170 comp rx=0 tx=0).stop 2024-03-16T20:06:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- 172.21.15.32:0/964764350 shutdown_connections 2024-03-16T20:06:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e015b2a0 0x7f30e015b6d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- 172.21.15.32:0/964764350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e00c81e0 0x7f30e00c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:46.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- 172.21.15.32:0/964764350 >> 172.21.15.32:0/964764350 conn(0x7f30e0070f90 msgr2=0x7f30e00713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:46.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- 172.21.15.32:0/964764350 shutdown_connections 2024-03-16T20:06:46.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- 172.21.15.32:0/964764350 wait complete. 2024-03-16T20:06:46.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 Processor -- start 2024-03-16T20:06:46.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- start start 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00c81e0 0x7f30e00c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e0151ef0 0x7f30e006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01f0930 con 0x7f30e015b2a0 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e014c0f0 con 0x7f30e00c81e0 2024-03-16T20:06:46.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e797f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e014c3d0 con 0x7f30e0151ef0 2024-03-16T20:06:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e5f1c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e4f1a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e0151ef0 0x7f30e006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e5f1c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51306/0 (socket says 172.21.15.32:51306) 2024-03-16T20:06:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.598+0000 7f30e5f1c640 1 -- 172.21.15.32:0/4274351190 learned_addr learned my addr 172.21.15.32:0/4274351190 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:46.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.602+0000 7f30e571b640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00c81e0 0x7f30e00c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.606+0000 7f30e5f1c640 1 -- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e0151ef0 msgr2=0x7f30e006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.606+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e0151ef0 0x7f30e006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.606+0000 7f30e5f1c640 1 -- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00c81e0 msgr2=0x7f30e00c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.606+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00c81e0 0x7f30e00c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.606+0000 7f30e5f1c640 1 -- 172.21.15.32:0/4274351190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d8009a70 con 0x7f30e015b2a0 2024-03-16T20:06:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.610+0000 7f30e5f1c640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f30d80b2040 tx=0x7f30d801bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8009450 con 0x7f30e015b2a0 2024-03-16T20:06:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d80b9410 con 0x7f30e015b2a0 2024-03-16T20:06:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80b97c0 con 0x7f30e015b2a0 2024-03-16T20:06:46.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e01f0270 con 0x7f30e015b2a0 2024-03-16T20:06:46.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e01eff20 con 0x7f30e015b2a0 2024-03-16T20:06:46.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.614+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e01f0a70 con 0x7f30e015b2a0 2024-03-16T20:06:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.650+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f30d80b6050 con 0x7f30e015b2a0 2024-03-16T20:06:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.654+0000 7f30d5ffb640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 0x7f30c4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.654+0000 7f30e571b640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 0x7f30c4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.654+0000 7f30e571b640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 0x7f30c4069330 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f30dc007fd0 tx=0x7f30dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.934+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f30d80f3b00 con 0x7f30e015b2a0 2024-03-16T20:06:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:46.934+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f30d80efda0 con 0x7f30e015b2a0 2024-03-16T20:06:47.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.154+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30e01f0a70 con 0x7f30e015b2a0 2024-03-16T20:06:47.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.170+0000 7f30d5ffb640 1 -- 172.21.15.32:0/4274351190 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f30e01f0a70 con 0x7f30e015b2a0 2024-03-16T20:06:47.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:47.159 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:47.159 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 msgr2=0x7f30c4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 0x7f30c4069330 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f30dc007fd0 tx=0x7f30dc009040 comp rx=0 tx=0).stop 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 msgr2=0x7f30e006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f30d80b2040 tx=0x7f30d801bf10 comp rx=0 tx=0).stop 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 shutdown_connections 2024-03-16T20:06:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30c4066e70 0x7f30c4069330 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e015b2a0 0x7f30e006cb70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30e0151ef0 0x7f30e006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 --2- 172.21.15.32:0/4274351190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00c81e0 0x7f30e00c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 >> 172.21.15.32:0/4274351190 conn(0x7f30e0070f90 msgr2=0x7f30e00cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 shutdown_connections 2024-03-16T20:06:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:47.174+0000 7f30e797f640 1 -- 172.21.15.32:0/4274351190 wait complete. 2024-03-16T20:06:47.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:47.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:48.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 Processor -- start 2024-03-16T20:06:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 -- start start 2024-03-16T20:06:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f819c071110 0x7f819c0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f819c0714f0 0x7f819c0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0ca1d0 con 0x7f819c0c2910 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c0c34a0 con 0x7f819c0714f0 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a274f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c0c9ba0 con 0x7f819c071110 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a0cec640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f819c0c2910 0x7f819c151a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:48550/0 (socket says 172.21.15.32:48550) 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f81a0cec640 1 -- 172.21.15.32:0/1399318301 learned_addr learned my addr 172.21.15.32:0/1399318301 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.910+0000 7f819b7fe640 1 --2- 172.21.15.32:0/1399318301 >> v2:172.21.15.32:3300/0 conn(0x7f819c0714f0 0x7f819c0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.914+0000 7f819bfff640 1 --2- 172.21.15.32:0/1399318301 >> v2:172.21.15.59:3300/0 conn(0x7f819c071110 0x7f819c0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.914+0000 7f819b7fe640 1 -- 172.21.15.32:0/1399318301 >> v2:172.21.15.59:3300/0 conn(0x7f819c071110 msgr2=0x7f819c0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.914+0000 7f819b7fe640 1 --2- 172.21.15.32:0/1399318301 >> v2:172.21.15.59:3300/0 conn(0x7f819c071110 0x7f819c0c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.914+0000 7f819b7fe640 1 -- 172.21.15.32:0/1399318301 >> v1:172.21.15.59:6789/0 conn(0x7f819c0c2910 legacy=0x7f819c151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.914+0000 7f819b7fe640 1 -- 172.21.15.32:0/1399318301 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c0c2ee0 con 0x7f819c0714f0 2024-03-16T20:06:48.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.918+0000 7f819b7fe640 1 --2- 172.21.15.32:0/1399318301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c0714f0 0x7f819c0ca740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f819c071a60 tx=0x7f819001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b747a18ea2870d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:48.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.922+0000 7f819affd640 1 -- 172.21.15.32:0/1399318301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81900a8970 con 0x7f819c0714f0 2024-03-16T20:06:48.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.922+0000 7f819affd640 1 -- 172.21.15.32:0/1399318301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81900a8ad0 con 0x7f819c0714f0 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.922+0000 7f819affd640 1 -- 172.21.15.32:0/1399318301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81900a8e60 con 0x7f819c0714f0 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- 172.21.15.32:0/1399318301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c0714f0 msgr2=0x7f819c0ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 --2- 172.21.15.32:0/1399318301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c0714f0 0x7f819c0ca740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f819c071a60 tx=0x7f819001ec70 comp rx=0 tx=0).stop 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- 172.21.15.32:0/1399318301 shutdown_connections 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 --2- 172.21.15.32:0/1399318301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c0714f0 0x7f819c0ca740 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 --2- 172.21.15.32:0/1399318301 >> v2:172.21.15.59:3300/0 conn(0x7f819c071110 0x7f819c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:48.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- 172.21.15.32:0/1399318301 >> 172.21.15.32:0/1399318301 conn(0x7f819c06afc0 msgr2=0x7f819c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:48.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- 172.21.15.32:0/1399318301 shutdown_connections 2024-03-16T20:06:48.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- 172.21.15.32:0/1399318301 wait complete. 2024-03-16T20:06:48.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 Processor -- start 2024-03-16T20:06:48.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.926+0000 7f81a274f640 1 -- start start 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f819c0714f0 0x7f819c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f819c0c2910 0x7f819c1f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c0c6f80 con 0x7f819c071110 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1550e0 con 0x7f819c0c2910 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a274f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1f1240 con 0x7f819c0714f0 2024-03-16T20:06:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f819bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f819bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51312/0 (socket says 172.21.15.32:51312) 2024-03-16T20:06:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f819bfff640 1 -- 172.21.15.32:0/2987738662 learned_addr learned my addr 172.21.15.32:0/2987738662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f81a0cec640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f819c0c2910 0x7f819c1f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.930+0000 7f819b7fe640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f819c0714f0 0x7f819c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:48.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 -- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f819c0714f0 msgr2=0x7f819c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:48.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f819c0714f0 0x7f819c0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:48.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 -- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f819c0c2910 msgr2=0x7f819c1f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:48.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f819c0c2910 0x7f819c1f16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:48.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 -- 172.21.15.32:0/2987738662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8190009420 con 0x7f819c071110 2024-03-16T20:06:48.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.934+0000 7f819bfff640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c820 tx=0x7f818c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c004220 con 0x7f819c071110 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0043a0 con 0x7f819c071110 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c004510 con 0x7f819c071110 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c0cb090 con 0x7f819c071110 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c155330 con 0x7f819c071110 2024-03-16T20:06:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:48.942+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c068000 con 0x7f819c071110 2024-03-16T20:06:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.230+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f818c007a70 con 0x7f819c071110 2024-03-16T20:06:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.230+0000 7f8198ff9640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 0x7f8188069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.230+0000 7f819b7fe640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 0x7f8188069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.230+0000 7f819b7fe640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 0x7f8188069300 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f81900b2000 tx=0x7f8190008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:49.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.246+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f818c055820 con 0x7f819c071110 2024-03-16T20:06:49.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.246+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f818c051ac0 con 0x7f819c071110 2024-03-16T20:06:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.618+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f819c0c7f20 con 0x7f819c071110 2024-03-16T20:06:49.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f8198ff9640 1 -- 172.21.15.32:0/2987738662 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f818c01e020 con 0x7f819c071110 2024-03-16T20:06:49.626 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:49.626 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 msgr2=0x7f8188069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 0x7f8188069300 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f81900b2000 tx=0x7f8190008040 comp rx=0 tx=0).stop 2024-03-16T20:06:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 msgr2=0x7f819c1f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c820 tx=0x7f818c00cce0 comp rx=0 tx=0).stop 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 shutdown_connections 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8188066e40 0x7f8188069300 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f819c0c2910 0x7f819c1f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f819c0714f0 0x7f819c0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 --2- 172.21.15.32:0/2987738662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f819c071110 0x7f819c1f3620 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 >> 172.21.15.32:0/2987738662 conn(0x7f819c06afc0 msgr2=0x7f819c0cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 shutdown_connections 2024-03-16T20:06:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:49.638+0000 7f81a274f640 1 -- 172.21.15.32:0/2987738662 wait complete. 2024-03-16T20:06:50.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:50.249 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:51.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 Processor -- start 2024-03-16T20:06:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 -- start start 2024-03-16T20:06:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c4151000 0x7f10c40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c40c36b0 0x7f10c415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40c8ec0 con 0x7f10c40c8980 2024-03-16T20:06:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c4072f10 con 0x7f10c4151000 2024-03-16T20:06:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10cbe0e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40c5000 con 0x7f10c40c36b0 2024-03-16T20:06:51.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10c9baa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c4151000 0x7f10c40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10c9baa640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c4151000 0x7f10c40c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35562/0 (socket says 172.21.15.32:35562) 2024-03-16T20:06:51.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10c9baa640 1 -- 172.21.15.32:0/3972237384 learned_addr learned my addr 172.21.15.32:0/3972237384 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:51.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.370+0000 7f10c93a9640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c40c36b0 0x7f10c415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.374+0000 7f10c9baa640 1 -- 172.21.15.32:0/3972237384 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c40c36b0 msgr2=0x7f10c415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.374+0000 7f10c9baa640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c40c36b0 0x7f10c415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.374+0000 7f10c9baa640 1 -- 172.21.15.32:0/3972237384 >> v1:172.21.15.32:6789/0 conn(0x7f10c40c8980 legacy=0x7f10c40c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.374+0000 7f10c9baa640 1 -- 172.21.15.32:0/3972237384 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4072950 con 0x7f10c4151000 2024-03-16T20:06:51.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.378+0000 7f10c9baa640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c4151000 0x7f10c40c8420 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009870 tx=0x7f10c001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb10bcec238b8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:51.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.382+0000 7f10c8ba8640 1 -- 172.21.15.32:0/3972237384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00aa640 con 0x7f10c4151000 2024-03-16T20:06:51.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.382+0000 7f10c8ba8640 1 -- 172.21.15.32:0/3972237384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c00aa7a0 con 0x7f10c4151000 2024-03-16T20:06:51.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.382+0000 7f10c8ba8640 1 -- 172.21.15.32:0/3972237384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00aab30 con 0x7f10c4151000 2024-03-16T20:06:51.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- 172.21.15.32:0/3972237384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c4151000 msgr2=0x7f10c40c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:51.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c4151000 0x7f10c40c8420 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009870 tx=0x7f10c001c520 comp rx=0 tx=0).stop 2024-03-16T20:06:51.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- 172.21.15.32:0/3972237384 shutdown_connections 2024-03-16T20:06:51.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c40c36b0 0x7f10c415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/3972237384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c4151000 0x7f10c40c8420 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- 172.21.15.32:0/3972237384 >> 172.21.15.32:0/3972237384 conn(0x7f10c406c0d0 msgr2=0x7f10c406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- 172.21.15.32:0/3972237384 shutdown_connections 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- 172.21.15.32:0/3972237384 wait complete. 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 Processor -- start 2024-03-16T20:06:51.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- start start 2024-03-16T20:06:51.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10c40c8980 0x7f10c41639a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c4151000 0x7f10c40cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41f6b30 con 0x7f10c40c36b0 2024-03-16T20:06:51.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c4162cd0 con 0x7f10c40c8980 2024-03-16T20:06:51.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10cbe0e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41f7020 con 0x7f10c4151000 2024-03-16T20:06:51.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10c9baa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10ca3ab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c4151000 0x7f10c40cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10c93a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10c40c8980 0x7f10c41639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10c9baa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35578/0 (socket says 172.21.15.32:35578) 2024-03-16T20:06:51.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.386+0000 7f10c9baa640 1 -- 172.21.15.32:0/178753693 learned_addr learned my addr 172.21.15.32:0/178753693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.390+0000 7f10c9baa640 1 -- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c4151000 msgr2=0x7f10c40cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.390+0000 7f10c9baa640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c4151000 0x7f10c40cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.390+0000 7f10c9baa640 1 -- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10c40c8980 msgr2=0x7f10c41639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.390+0000 7f10c9baa640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10c40c8980 0x7f10c41639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.390+0000 7f10c9baa640 1 -- 172.21.15.32:0/178753693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00094d0 con 0x7f10c40c36b0 2024-03-16T20:06:51.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.394+0000 7f10c9baa640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f10c001cdc0 tx=0x7f10c001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.398+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00bd5f0 con 0x7f10c40c36b0 2024-03-16T20:06:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.398+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c00bd770 con 0x7f10c40c36b0 2024-03-16T20:06:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.398+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c41f6d40 con 0x7f10c40c36b0 2024-03-16T20:06:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.398+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00bdb20 con 0x7f10c40c36b0 2024-03-16T20:06:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.398+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c4162960 con 0x7f10c40c36b0 2024-03-16T20:06:51.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.402+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c40c90e0 con 0x7f10c40c36b0 2024-03-16T20:06:51.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.438+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f10c00a8050 con 0x7f10c40c36b0 2024-03-16T20:06:51.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.438+0000 7f10ba7fc640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 0x7f10a8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:51.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.438+0000 7f10c93a9640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 0x7f10a8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:51.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.438+0000 7f10c93a9640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 0x7f10a8069300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f10c406cae0 tx=0x7f10b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:51.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.442+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f10c00f0100 con 0x7f10c40c36b0 2024-03-16T20:06:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.726+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f10c00ec3a0 con 0x7f10c40c36b0 2024-03-16T20:06:51.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.974+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10c4068000 con 0x7f10c40c36b0 2024-03-16T20:06:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10ba7fc640 1 -- 172.21.15.32:0/178753693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f10c00aa600 con 0x7f10c40c36b0 2024-03-16T20:06:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:51.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 msgr2=0x7f10a8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 0x7f10a8069300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f10c406cae0 tx=0x7f10b4009190 comp rx=0 tx=0).stop 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 msgr2=0x7f10c4163460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f10c001cdc0 tx=0x7f10c001ad60 comp rx=0 tx=0).stop 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 shutdown_connections 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f10a8066e40 0x7f10a8069300 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f10c4151000 0x7f10c40cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10c40c8980 0x7f10c41639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 --2- 172.21.15.32:0/178753693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c40c36b0 0x7f10c4163460 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 >> 172.21.15.32:0/178753693 conn(0x7f10c406c0d0 msgr2=0x7f10c414dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 shutdown_connections 2024-03-16T20:06:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:51.986+0000 7f10cbe0e640 1 -- 172.21.15.32:0/178753693 wait complete. 2024-03-16T20:06:52.595 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:52.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:53.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:53.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 Processor -- start 2024-03-16T20:06:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 -- start start 2024-03-16T20:06:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4e40725e0 0x7fd4e4073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4151020 con 0x7fd4e4150c40 2024-03-16T20:06:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4072da0 con 0x7fd4e415a420 2024-03-16T20:06:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4eac16640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e415a000 con 0x7fd4e40725e0 2024-03-16T20:06:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4e3fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4e40725e0 0x7fd4e4073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4e89b2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd4e4150c40 0x7fd4e4070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41792/0 (socket says 172.21.15.32:41792) 2024-03-16T20:06:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.718+0000 7fd4e89b2640 1 -- 172.21.15.32:0/3172130128 learned_addr learned my addr 172.21.15.32:0/3172130128 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.722+0000 7fd4e3fff640 1 -- 172.21.15.32:0/3172130128 >> v1:172.21.15.59:6789/0 conn(0x7fd4e415a420 legacy=0x7fd4e415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.722+0000 7fd4e3fff640 1 -- 172.21.15.32:0/3172130128 >> v1:172.21.15.32:6789/0 conn(0x7fd4e4150c40 legacy=0x7fd4e4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.722+0000 7fd4e3fff640 1 -- 172.21.15.32:0/3172130128 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e415e310 con 0x7fd4e40725e0 2024-03-16T20:06:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.730+0000 7fd4e3fff640 1 --2- 172.21.15.32:0/3172130128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e40725e0 0x7fd4e4073070 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00097c0 tx=0x7fd4d001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b287bc52909d6baa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4e37fe640 1 -- 172.21.15.32:0/3172130128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00a6040 con 0x7fd4e40725e0 2024-03-16T20:06:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4e37fe640 1 -- 172.21.15.32:0/3172130128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d00aa600 con 0x7fd4e40725e0 2024-03-16T20:06:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4e37fe640 1 -- 172.21.15.32:0/3172130128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00aa990 con 0x7fd4e40725e0 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- 172.21.15.32:0/3172130128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e40725e0 msgr2=0x7fd4e4073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 --2- 172.21.15.32:0/3172130128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e40725e0 0x7fd4e4073070 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00097c0 tx=0x7fd4d001c4a0 comp rx=0 tx=0).stop 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- 172.21.15.32:0/3172130128 shutdown_connections 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 --2- 172.21.15.32:0/3172130128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e40725e0 0x7fd4e4073070 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- 172.21.15.32:0/3172130128 >> 172.21.15.32:0/3172130128 conn(0x7fd4e406c0d0 msgr2=0x7fd4e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- 172.21.15.32:0/3172130128 shutdown_connections 2024-03-16T20:06:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- 172.21.15.32:0/3172130128 wait complete. 2024-03-16T20:06:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 Processor -- start 2024-03-16T20:06:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.734+0000 7fd4eac16640 1 -- start start 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e40725e0 0x7fd4e4145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd4e415a420 0x7fd4e414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40cc6b0 con 0x7fd4e4150c40 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40cd230 con 0x7fd4e40725e0 2024-03-16T20:06:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4eac16640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41458a0 con 0x7fd4e415a420 2024-03-16T20:06:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35586/0 (socket says 172.21.15.32:35586) 2024-03-16T20:06:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4e3fff640 1 -- 172.21.15.32:0/595067888 learned_addr learned my addr 172.21.15.32:0/595067888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4e89b2640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e40725e0 0x7fd4e4145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.738+0000 7fd4e91b3640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd4e415a420 0x7fd4e414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.742+0000 7fd4e3fff640 1 -- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd4e415a420 msgr2=0x7fd4e414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.742+0000 7fd4e3fff640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd4e415a420 0x7fd4e414b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.742+0000 7fd4e3fff640 1 -- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e40725e0 msgr2=0x7fd4e4145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.742+0000 7fd4e3fff640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e40725e0 0x7fd4e4145360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.742+0000 7fd4e3fff640 1 -- 172.21.15.32:0/595067888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0009420 con 0x7fd4e4150c40 2024-03-16T20:06:53.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.746+0000 7fd4e3fff640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00097c0 tx=0x7fd4d001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00b3800 con 0x7fd4e4150c40 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d00b3980 con 0x7fd4e4150c40 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e415d560 con 0x7fd4e4150c40 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00b3d30 con 0x7fd4e4150c40 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e40cc900 con 0x7fd4e4150c40 2024-03-16T20:06:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.750+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e4068000 con 0x7fd4e4150c40 2024-03-16T20:06:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.770+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd4d00c7410 con 0x7fd4e4150c40 2024-03-16T20:06:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.774+0000 7fd4e17fa640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 0x7fd4c8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.774+0000 7fd4e89b2640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 0x7fd4c8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:53.774+0000 7fd4e89b2640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 0x7fd4c8069300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8000f70 tx=0x7fd4d8007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.078+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd4d00eff00 con 0x7fd4e4150c40 2024-03-16T20:06:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.078+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd4d00ec1a0 con 0x7fd4e4150c40 2024-03-16T20:06:54.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.278+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e4143880 con 0x7fd4e4150c40 2024-03-16T20:06:54.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.302+0000 7fd4e17fa640 1 -- 172.21.15.32:0/595067888 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fd4d00ef860 con 0x7fd4e4150c40 2024-03-16T20:06:54.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:54.291 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 msgr2=0x7fd4c8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 0x7fd4c8069300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8000f70 tx=0x7fd4d8007e90 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 msgr2=0x7fd4e414c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00097c0 tx=0x7fd4d001ace0 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 shutdown_connections 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd4c8066e40 0x7fd4c8069300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd4e415a420 0x7fd4e414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4e4150c40 0x7fd4e414c7c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 --2- 172.21.15.32:0/595067888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e40725e0 0x7fd4e4145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 >> 172.21.15.32:0/595067888 conn(0x7fd4e406c0d0 msgr2=0x7fd4e40cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 shutdown_connections 2024-03-16T20:06:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:54.306+0000 7fd4eac16640 1 -- 172.21.15.32:0/595067888 wait complete. 2024-03-16T20:06:54.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:54.920 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:55.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:56.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 Processor -- start 2024-03-16T20:06:56.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 -- start start 2024-03-16T20:06:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80e806ea80 0x7f80e81587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e80c68e0 con 0x7f80e80b1eb0 2024-03-16T20:06:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e8067340 con 0x7f80e806ea80 2024-03-16T20:06:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80f00b2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e806c1f0 con 0x7f80e80c7ce0 2024-03-16T20:06:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.042+0000 7f80ee64f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80e806ea80 0x7f80e81587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80e806ea80 0x7f80e81587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35588/0 (socket says 172.21.15.32:35588) 2024-03-16T20:06:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 -- 172.21.15.32:0/4100074108 learned_addr learned my addr 172.21.15.32:0/4100074108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:56.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ed64d640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 -- 172.21.15.32:0/4100074108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 msgr2=0x7f80e8158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 -- 172.21.15.32:0/4100074108 >> v1:172.21.15.59:6789/0 conn(0x7f80e80b1eb0 legacy=0x7f80e8072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.046+0000 7f80ee64f640 1 -- 172.21.15.32:0/4100074108 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e806c4d0 con 0x7f80e806ea80 2024-03-16T20:06:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.050+0000 7f80ee64f640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e81587b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009e50 tx=0x7f80e001f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99d654db25a04ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80ece4c640 1 -- 172.21.15.32:0/4100074108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00af400 con 0x7f80e806ea80 2024-03-16T20:06:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80ece4c640 1 -- 172.21.15.32:0/4100074108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e00ab050 con 0x7f80e806ea80 2024-03-16T20:06:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80ece4c640 1 -- 172.21.15.32:0/4100074108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00af780 con 0x7f80e806ea80 2024-03-16T20:06:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- 172.21.15.32:0/4100074108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 msgr2=0x7f80e81587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e81587b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009e50 tx=0x7f80e001f210 comp rx=0 tx=0).stop 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- 172.21.15.32:0/4100074108 shutdown_connections 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e81587b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 --2- 172.21.15.32:0/4100074108 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- 172.21.15.32:0/4100074108 >> 172.21.15.32:0/4100074108 conn(0x7f80e806afc0 msgr2=0x7f80e806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- 172.21.15.32:0/4100074108 shutdown_connections 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- 172.21.15.32:0/4100074108 wait complete. 2024-03-16T20:06:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 Processor -- start 2024-03-16T20:06:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.058+0000 7f80f00b2640 1 -- start start 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80e80b1eb0 0x7f80e814fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8157670 con 0x7f80e806ea80 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e814ffb0 con 0x7f80e80b1eb0 2024-03-16T20:06:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80f00b2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81edf70 con 0x7f80e80c7ce0 2024-03-16T20:06:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80ede4e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80ed64d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80e80b1eb0 0x7f80e814fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80ee64f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80ede4e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35594/0 (socket says 172.21.15.32:35594) 2024-03-16T20:06:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.062+0000 7f80ede4e640 1 -- 172.21.15.32:0/2907716245 learned_addr learned my addr 172.21.15.32:0/2907716245 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.066+0000 7f80ede4e640 1 -- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 msgr2=0x7f80e8150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.066+0000 7f80ede4e640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.066+0000 7f80ede4e640 1 -- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80e80b1eb0 msgr2=0x7f80e814fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.066+0000 7f80ede4e640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80e80b1eb0 0x7f80e814fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.066+0000 7f80ede4e640 1 -- 172.21.15.32:0/2907716245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0009b30 con 0x7f80e806ea80 2024-03-16T20:06:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.070+0000 7f80ede4e640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c530 tx=0x7f80e400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.074+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4005220 con 0x7f80e806ea80 2024-03-16T20:06:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.074+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4005380 con 0x7f80e806ea80 2024-03-16T20:06:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.074+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8151430 con 0x7f80e806ea80 2024-03-16T20:06:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.078+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4007270 con 0x7f80e806ea80 2024-03-16T20:06:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.078+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81504b0 con 0x7f80e806ea80 2024-03-16T20:06:56.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.078+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8068000 con 0x7f80e806ea80 2024-03-16T20:06:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.102+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f80e40073d0 con 0x7f80e806ea80 2024-03-16T20:06:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.102+0000 7f80de7fc640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 0x7f80cc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.102+0000 7f80ed64d640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 0x7f80cc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.102+0000 7f80ed64d640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 0x7f80cc069380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f80e80c7b50 tx=0x7f80d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.390+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f80e4007680 con 0x7f80e806ea80 2024-03-16T20:06:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.390+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f80e4051e40 con 0x7f80e806ea80 2024-03-16T20:06:56.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.598+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80e814e240 con 0x7f80e806ea80 2024-03-16T20:06:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.614+0000 7f80de7fc640 1 -- 172.21.15.32:0/2907716245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f80e4055500 con 0x7f80e806ea80 2024-03-16T20:06:56.604 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:56.604 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 msgr2=0x7f80cc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 0x7f80cc069380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f80e80c7b50 tx=0x7f80d8008040 comp rx=0 tx=0).stop 2024-03-16T20:06:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 msgr2=0x7f80e814d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c530 tx=0x7f80e400c9f0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 shutdown_connections 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80cc066ec0 0x7f80cc069380 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f80e80c7ce0 0x7f80e8150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80e80b1eb0 0x7f80e814fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 --2- 172.21.15.32:0/2907716245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e806ea80 0x7f80e814d8d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 >> 172.21.15.32:0/2907716245 conn(0x7f80e806afc0 msgr2=0x7f80e80c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 shutdown_connections 2024-03-16T20:06:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:56.618+0000 7f80f00b2640 1 -- 172.21.15.32:0/2907716245 wait complete. 2024-03-16T20:06:57.231 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:57.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:06:58.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:06:58.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 Processor -- start 2024-03-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 -- start start 2024-03-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7080153fa0 0x7f7080070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f7080072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70800c8590 con 0x7f70800c7470 2024-03-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800c3f00 con 0x7f7080153fa0 2024-03-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7088155640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7080070650 con 0x7f70800724a0 2024-03-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.354+0000 7f7085ef1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7080153fa0 0x7f7080070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.358+0000 7f70856f0640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f70800c7470 0x7f708015d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41794/0 (socket says 172.21.15.32:41794) 2024-03-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.358+0000 7f70856f0640 1 -- 172.21.15.32:0/2804403000 learned_addr learned my addr 172.21.15.32:0/2804403000 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.358+0000 7f70866f2640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f7080072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.362+0000 7f7085ef1640 1 -- 172.21.15.32:0/2804403000 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 msgr2=0x7f7080072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:06:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.362+0000 7f7085ef1640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f7080072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.362+0000 7f7085ef1640 1 -- 172.21.15.32:0/2804403000 >> v1:172.21.15.32:6789/0 conn(0x7f70800c7470 legacy=0x7f708015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.362+0000 7f7085ef1640 1 -- 172.21.15.32:0/2804403000 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080073590 con 0x7f7080153fa0 2024-03-16T20:06:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.366+0000 7f7085ef1640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080070840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f707c0097c0 tx=0x7f707c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7606f50dad6ac15c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7084eef640 1 -- 172.21.15.32:0/2804403000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c0a8990 con 0x7f7080153fa0 2024-03-16T20:06:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7084eef640 1 -- 172.21.15.32:0/2804403000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c0a8af0 con 0x7f7080153fa0 2024-03-16T20:06:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7084eef640 1 -- 172.21.15.32:0/2804403000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c0a8e80 con 0x7f7080153fa0 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 -- 172.21.15.32:0/2804403000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 msgr2=0x7f7080070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080070840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f707c0097c0 tx=0x7f707c01ec90 comp rx=0 tx=0).stop 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 -- 172.21.15.32:0/2804403000 shutdown_connections 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f7080072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 --2- 172.21.15.32:0/2804403000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080070840 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 -- 172.21.15.32:0/2804403000 >> 172.21.15.32:0/2804403000 conn(0x7f708006c0d0 msgr2=0x7f708006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 -- 172.21.15.32:0/2804403000 shutdown_connections 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.374+0000 7f7088155640 1 -- 172.21.15.32:0/2804403000 wait complete. 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 Processor -- start 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 -- start start 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f70800cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70800c7470 0x7f70800ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080155650 con 0x7f7080153fa0 2024-03-16T20:06:58.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080159250 con 0x7f70800c7470 2024-03-16T20:06:58.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.378+0000 7f7088155640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708013daf0 con 0x7f70800724a0 2024-03-16T20:06:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.382+0000 7f7085ef1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f70800cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.382+0000 7f70866f2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.382+0000 7f70866f2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35614/0 (socket says 172.21.15.32:35614) 2024-03-16T20:06:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.382+0000 7f70866f2640 1 -- 172.21.15.32:0/2389938433 learned_addr learned my addr 172.21.15.32:0/2389938433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:06:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.382+0000 7f70856f0640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70800c7470 0x7f70800ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.386+0000 7f70866f2640 1 -- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 msgr2=0x7f70800cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:58.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.386+0000 7f70866f2640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f70800cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:58.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.386+0000 7f70866f2640 1 -- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70800c7470 msgr2=0x7f70800ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:58.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.386+0000 7f70866f2640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70800c7470 0x7f70800ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:58.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.386+0000 7f70866f2640 1 -- 172.21.15.32:0/2389938433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c009420 con 0x7f7080153fa0 2024-03-16T20:06:58.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.390+0000 7f70866f2640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f707800efc0 tx=0x7f707800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800ced0 con 0x7f7080153fa0 2024-03-16T20:06:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707800f040 con 0x7f7080153fa0 2024-03-16T20:06:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780146c0 con 0x7f7080153fa0 2024-03-16T20:06:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70801552e0 con 0x7f7080153fa0 2024-03-16T20:06:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7080159730 con 0x7f7080153fa0 2024-03-16T20:06:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.402+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7080155790 con 0x7f7080153fa0 2024-03-16T20:06:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.746+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7078014820 con 0x7f7080153fa0 2024-03-16T20:06:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.746+0000 7f70767fc640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 0x7f7064069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:06:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.746+0000 7f7085ef1640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 0x7f7064069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:06:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.746+0000 7f7085ef1640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 0x7f7064069350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f707c0b2000 tx=0x7f707c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:06:58.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.766+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7078056640 con 0x7f7080153fa0 2024-03-16T20:06:58.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.766+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f70780528e0 con 0x7f7080153fa0 2024-03-16T20:06:58.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:58.994+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7080155790 con 0x7f7080153fa0 2024-03-16T20:06:59.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.034+0000 7f70767fc640 1 -- 172.21.15.32:0/2389938433 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f7080155790 con 0x7f7080153fa0 2024-03-16T20:06:59.025 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:06:59.025 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:06:59.025 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:06:59.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 msgr2=0x7f7064069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:59.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 0x7f7064069350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f707c0b2000 tx=0x7f707c008040 comp rx=0 tx=0).stop 2024-03-16T20:06:59.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 msgr2=0x7f7080155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:06:59.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f707800efc0 tx=0x7f707800c490 comp rx=0 tx=0).stop 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 shutdown_connections 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7064066e90 0x7f7064069350 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7080153fa0 0x7f7080155a70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70800c7470 0x7f70800ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 --2- 172.21.15.32:0/2389938433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f70800724a0 0x7f70800cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 >> 172.21.15.32:0/2389938433 conn(0x7f708006c0d0 msgr2=0x7f708006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 shutdown_connections 2024-03-16T20:06:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:06:59.038+0000 7f7088155640 1 -- 172.21.15.32:0/2389938433 wait complete. 2024-03-16T20:06:59.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:06:59.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:00.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 Processor -- start 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 -- start start 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02e8151000 0x7f02e80c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02e80c8980 0x7f02e815a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80c8ec0 con 0x7f02e80c36b0 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e8072f10 con 0x7f02e80c8980 2024-03-16T20:07:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02ee125640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e80c5000 con 0x7f02e8151000 2024-03-16T20:07:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.766+0000 7f02e77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02e8151000 0x7f02e80c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.770+0000 7f02e77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02e8151000 0x7f02e80c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:49870/0 (socket says 172.21.15.32:49870) 2024-03-16T20:07:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.770+0000 7f02e77fe640 1 -- 172.21.15.32:0/2982454427 learned_addr learned my addr 172.21.15.32:0/2982454427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:00.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.774+0000 7f02e77fe640 1 -- 172.21.15.32:0/2982454427 >> v2:172.21.15.32:3300/0 conn(0x7f02e80c8980 msgr2=0x7f02e815a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:00.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.774+0000 7f02e77fe640 1 --2- 172.21.15.32:0/2982454427 >> v2:172.21.15.32:3300/0 conn(0x7f02e80c8980 0x7f02e815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.774+0000 7f02e77fe640 1 -- 172.21.15.32:0/2982454427 >> v1:172.21.15.59:6789/0 conn(0x7f02e80c36b0 legacy=0x7f02e80c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:00.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.774+0000 7f02e77fe640 1 -- 172.21.15.32:0/2982454427 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8072950 con 0x7f02e8151000 2024-03-16T20:07:00.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.778+0000 7f02e77fe640 1 --2- 172.21.15.32:0/2982454427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e8151000 0x7f02e80c8420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02e8070c10 tx=0x7f02d801c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa8cf6cf8f0f599e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:00.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02e67fc640 1 -- 172.21.15.32:0/2982454427 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d80aa640 con 0x7f02e8151000 2024-03-16T20:07:00.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02e67fc640 1 -- 172.21.15.32:0/2982454427 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d80aa7a0 con 0x7f02e8151000 2024-03-16T20:07:00.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02e67fc640 1 -- 172.21.15.32:0/2982454427 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d80aab30 con 0x7f02e8151000 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 -- 172.21.15.32:0/2982454427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e8151000 msgr2=0x7f02e80c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 --2- 172.21.15.32:0/2982454427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e8151000 0x7f02e80c8420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02e8070c10 tx=0x7f02d801c4e0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 -- 172.21.15.32:0/2982454427 shutdown_connections 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 --2- 172.21.15.32:0/2982454427 >> v2:172.21.15.32:3300/0 conn(0x7f02e80c8980 0x7f02e815a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 --2- 172.21.15.32:0/2982454427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e8151000 0x7f02e80c8420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 -- 172.21.15.32:0/2982454427 >> 172.21.15.32:0/2982454427 conn(0x7f02e806c0d0 msgr2=0x7f02e806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 -- 172.21.15.32:0/2982454427 shutdown_connections 2024-03-16T20:07:00.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.786+0000 7f02ee125640 1 -- 172.21.15.32:0/2982454427 wait complete. 2024-03-16T20:07:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 Processor -- start 2024-03-16T20:07:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 -- start start 2024-03-16T20:07:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 0x7f02e81f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 0x7f02e806b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e8153d10 0x7f02e81565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8154760 con 0x7f02e80c8980 2024-03-16T20:07:00.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8154480 con 0x7f02e80c36b0 2024-03-16T20:07:00.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02ee125640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8154ce0 con 0x7f02e8153d10 2024-03-16T20:07:00.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02e6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 0x7f02e806b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:00.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 0x7f02e81f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:00.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.790+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e8153d10 0x7f02e81565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:00.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.794+0000 7f02e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 0x7f02e81f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:49874/0 (socket says 172.21.15.32:49874) 2024-03-16T20:07:00.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.794+0000 7f02e77fe640 1 -- 172.21.15.32:0/2005061816 learned_addr learned my addr 172.21.15.32:0/2005061816 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:00.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.794+0000 7f02e6ffd640 1 -- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e8153d10 msgr2=0x7f02e81565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:00.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.794+0000 7f02e6ffd640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e8153d10 0x7f02e81565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.798+0000 7f02e6ffd640 1 -- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 msgr2=0x7f02e81f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:00.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.798+0000 7f02e6ffd640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 0x7f02e81f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:00.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.798+0000 7f02e6ffd640 1 -- 172.21.15.32:0/2005061816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80094d0 con 0x7f02e80c8980 2024-03-16T20:07:00.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.798+0000 7f02e6ffd640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 0x7f02e806b220 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f02e80736e0 tx=0x7f02dc00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.802+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc003600 con 0x7f02e80c8980 2024-03-16T20:07:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.802+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc003760 con 0x7f02e80c8980 2024-03-16T20:07:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.802+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e8154fc0 con 0x7f02e80c8980 2024-03-16T20:07:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.802+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc003b30 con 0x7f02e80c8980 2024-03-16T20:07:00.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.802+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e814d570 con 0x7f02e80c8980 2024-03-16T20:07:00.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:00.806+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8068000 con 0x7f02e80c8980 2024-03-16T20:07:01.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.094+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02dc003c90 con 0x7f02e80c8980 2024-03-16T20:07:01.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.094+0000 7f02cffff640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 0x7f02d4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:01.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.094+0000 7f02e77fe640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 0x7f02d4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.094+0000 7f02e77fe640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 0x7f02d4069350 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f02d801c520 tx=0x7f02d80b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:01.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.118+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f02dc0553c0 con 0x7f02e80c8980 2024-03-16T20:07:01.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.118+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f02dc051660 con 0x7f02e80c8980 2024-03-16T20:07:01.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.438+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e8073410 con 0x7f02e80c8980 2024-03-16T20:07:01.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.462+0000 7f02cffff640 1 -- 172.21.15.32:0/2005061816 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f02dc017090 con 0x7f02e80c8980 2024-03-16T20:07:01.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:01.451 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:01.451 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 msgr2=0x7f02d4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 0x7f02d4069350 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f02d801c520 tx=0x7f02d80b5040 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 msgr2=0x7f02e806b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 0x7f02e806b220 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f02e80736e0 tx=0x7f02dc00c870 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 shutdown_connections 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02d4066e90 0x7f02d4069350 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02e8153d10 0x7f02e81565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02e80c8980 0x7f02e806b220 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 --2- 172.21.15.32:0/2005061816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02e80c36b0 0x7f02e81f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 >> 172.21.15.32:0/2005061816 conn(0x7f02e806c0d0 msgr2=0x7f02e80caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 shutdown_connections 2024-03-16T20:07:01.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:01.466+0000 7f02ee125640 1 -- 172.21.15.32:0/2005061816 wait complete. 2024-03-16T20:07:02.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:02.075 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:03.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.198+0000 7f37ceae5640 1 Processor -- start 2024-03-16T20:07:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.198+0000 7f37ceae5640 1 -- start start 2024-03-16T20:07:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37ceae5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f37c8151000 0x7f37c80c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37ceae5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c80c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37ceae5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c80c7a40 con 0x7f37c8156b30 2024-03-16T20:07:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37ceae5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80c52e0 con 0x7f37c8151000 2024-03-16T20:07:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37ceae5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80c9270 con 0x7f37c80c81b0 2024-03-16T20:07:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37cc881640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f37c8151000 0x7f37c80c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37cd082640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c80c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37cd082640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c80c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:45066/0 (socket says 172.21.15.32:45066) 2024-03-16T20:07:03.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.202+0000 7f37cd082640 1 -- 172.21.15.32:0/1765692580 learned_addr learned my addr 172.21.15.32:0/1765692580 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:03.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.206+0000 7f37cc881640 1 -- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 msgr2=0x7f37c80c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:03.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.206+0000 7f37cc881640 1 --2- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c80c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.206+0000 7f37cc881640 1 -- 172.21.15.32:0/1765692580 >> v1:172.21.15.59:6789/0 conn(0x7f37c8156b30 legacy=0x7f37c80c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.206+0000 7f37cc881640 1 -- 172.21.15.32:0/1765692580 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80c9510 con 0x7f37c8151000 2024-03-16T20:07:03.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.218+0000 7f37cc881640 1 --2- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8151000 0x7f37c80c70d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0097c0 tx=0x7f37bc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=749fa231e6263853 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:03.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.222+0000 7f37c77fe640 1 -- 172.21.15.32:0/1765692580 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0a8990 con 0x7f37c8151000 2024-03-16T20:07:03.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37c77fe640 1 -- 172.21.15.32:0/1765692580 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc0a8af0 con 0x7f37c8151000 2024-03-16T20:07:03.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37c77fe640 1 -- 172.21.15.32:0/1765692580 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0a8e80 con 0x7f37c8151000 2024-03-16T20:07:03.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8151000 msgr2=0x7f37c80c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8151000 0x7f37c80c70d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0097c0 tx=0x7f37bc01ec90 comp rx=0 tx=0).stop 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- 172.21.15.32:0/1765692580 shutdown_connections 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c80c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- 172.21.15.32:0/1765692580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8151000 0x7f37c80c70d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- 172.21.15.32:0/1765692580 >> 172.21.15.32:0/1765692580 conn(0x7f37c806c0d0 msgr2=0x7f37c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- 172.21.15.32:0/1765692580 shutdown_connections 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- 172.21.15.32:0/1765692580 wait complete. 2024-03-16T20:07:03.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 Processor -- start 2024-03-16T20:07:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 -- start start 2024-03-16T20:07:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c814c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 0x7f37c81e4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.226+0000 7f37ceae5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 0x7f37c8146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37ceae5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c814d230 con 0x7f37c8151000 2024-03-16T20:07:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37ceae5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c814d410 con 0x7f37c8156b30 2024-03-16T20:07:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37ceae5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c81f2fa0 con 0x7f37c80c81b0 2024-03-16T20:07:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37cd082640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 0x7f37c8146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37c7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 0x7f37c81e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37cc881640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c814c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37cd082640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 0x7f37c8146f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:49882/0 (socket says 172.21.15.32:49882) 2024-03-16T20:07:03.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.230+0000 7f37cd082640 1 -- 172.21.15.32:0/2640986076 learned_addr learned my addr 172.21.15.32:0/2640986076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:03.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.234+0000 7f37c7fff640 1 -- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 msgr2=0x7f37c814c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.234+0000 7f37c7fff640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c814c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.234+0000 7f37c7fff640 1 -- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 msgr2=0x7f37c8146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.234+0000 7f37c7fff640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 0x7f37c8146f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.234+0000 7f37c7fff640 1 -- 172.21.15.32:0/2640986076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc009420 con 0x7f37c8151000 2024-03-16T20:07:03.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.238+0000 7f37c7fff640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 0x7f37c81e4570 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c3c0 tx=0x7f37b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:03.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40052d0 con 0x7f37c8151000 2024-03-16T20:07:03.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c81f3280 con 0x7f37c8151000 2024-03-16T20:07:03.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c81f3840 con 0x7f37c8151000 2024-03-16T20:07:03.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c8068000 con 0x7f37c8151000 2024-03-16T20:07:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4005430 con 0x7f37c8151000 2024-03-16T20:07:03.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.242+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4007240 con 0x7f37c8151000 2024-03-16T20:07:03.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.526+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f37b40073a0 con 0x7f37c8151000 2024-03-16T20:07:03.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.526+0000 7f37c57fa640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 0x7f37ac069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:03.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.526+0000 7f37cc881640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 0x7f37ac069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:03.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.526+0000 7f37cc881640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 0x7f37ac069330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0b2000 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:03.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.550+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f37b4007650 con 0x7f37c8151000 2024-03-16T20:07:03.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.550+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f37b4051370 con 0x7f37c8151000 2024-03-16T20:07:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.850+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c8073430 con 0x7f37c8151000 2024-03-16T20:07:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.870+0000 7f37c57fa640 1 -- 172.21.15.32:0/2640986076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f37b4020070 con 0x7f37c8151000 2024-03-16T20:07:03.860 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:03.860 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 msgr2=0x7f37ac069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 0x7f37ac069330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0b2000 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 msgr2=0x7f37c81e4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 0x7f37c81e4570 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c3c0 tx=0x7f37b400c880 comp rx=0 tx=0).stop 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 shutdown_connections 2024-03-16T20:07:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37ac066e70 0x7f37ac069330 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c8156b30 0x7f37c8146f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37c8151000 0x7f37c81e4570 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 --2- 172.21.15.32:0/2640986076 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f37c80c81b0 0x7f37c814c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 >> 172.21.15.32:0/2640986076 conn(0x7f37c806c0d0 msgr2=0x7f37c8072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 shutdown_connections 2024-03-16T20:07:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:03.874+0000 7f37ceae5640 1 -- 172.21.15.32:0/2640986076 wait complete. 2024-03-16T20:07:04.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:04.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:05.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:05.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 Processor -- start 2024-03-16T20:07:05.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 -- start start 2024-03-16T20:07:05.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b2006f5a0 0x7f9b2015a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b2006f980 0x7f9b2006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2015ae90 con 0x7f9b200c8c40 2024-03-16T20:07:05.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2006dae0 con 0x7f9b2006f980 2024-03-16T20:07:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b25d8a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b200cd4e0 con 0x7f9b2006f5a0 2024-03-16T20:07:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b1effd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b2006f980 0x7f9b2006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b1effd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b2006f980 0x7f9b2006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40372/0 (socket says 172.21.15.32:40372) 2024-03-16T20:07:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b1effd640 1 -- 172.21.15.32:0/3829181155 learned_addr learned my addr 172.21.15.32:0/3829181155 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.614+0000 7f9b1f7fe640 1 --2- 172.21.15.32:0/3829181155 >> v2:172.21.15.59:3300/0 conn(0x7f9b2006f5a0 0x7f9b2015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.618+0000 7f9b1effd640 1 -- 172.21.15.32:0/3829181155 >> v2:172.21.15.59:3300/0 conn(0x7f9b2006f5a0 msgr2=0x7f9b2015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.618+0000 7f9b1effd640 1 --2- 172.21.15.32:0/3829181155 >> v2:172.21.15.59:3300/0 conn(0x7f9b2006f5a0 0x7f9b2015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.618+0000 7f9b1effd640 1 -- 172.21.15.32:0/3829181155 >> v1:172.21.15.59:6789/0 conn(0x7f9b200c8c40 legacy=0x7f9b20153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.618+0000 7f9b1effd640 1 -- 172.21.15.32:0/3829181155 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b200c8140 con 0x7f9b2006f980 2024-03-16T20:07:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.622+0000 7f9b1effd640 1 --2- 172.21.15.32:0/3829181155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b2006f980 0x7f9b2006c0d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9b200ce600 tx=0x7f9b1401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38fa08926adec19 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.626+0000 7f9b1e7fc640 1 -- 172.21.15.32:0/3829181155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b140a8a10 con 0x7f9b2006f980 2024-03-16T20:07:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.626+0000 7f9b1e7fc640 1 -- 172.21.15.32:0/3829181155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b140a8b70 con 0x7f9b2006f980 2024-03-16T20:07:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.626+0000 7f9b1e7fc640 1 -- 172.21.15.32:0/3829181155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b140b0440 con 0x7f9b2006f980 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- 172.21.15.32:0/3829181155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b2006f980 msgr2=0x7f9b2006c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 --2- 172.21.15.32:0/3829181155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b2006f980 0x7f9b2006c0d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9b200ce600 tx=0x7f9b1401ed10 comp rx=0 tx=0).stop 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- 172.21.15.32:0/3829181155 shutdown_connections 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 --2- 172.21.15.32:0/3829181155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b2006f980 0x7f9b2006c0d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 --2- 172.21.15.32:0/3829181155 >> v2:172.21.15.59:3300/0 conn(0x7f9b2006f5a0 0x7f9b2015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- 172.21.15.32:0/3829181155 >> 172.21.15.32:0/3829181155 conn(0x7f9b20066cd0 msgr2=0x7f9b200670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- 172.21.15.32:0/3829181155 shutdown_connections 2024-03-16T20:07:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- 172.21.15.32:0/3829181155 wait complete. 2024-03-16T20:07:05.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 Processor -- start 2024-03-16T20:07:05.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.630+0000 7f9b25d8a640 1 -- start start 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b2006f5a0 0x7f9b20157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b2006f980 0x7f9b20149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2015bb50 con 0x7f9b200c8c40 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2014a330 con 0x7f9b2006f5a0 2024-03-16T20:07:05.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b25d8a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2015cc20 con 0x7f9b2006f980 2024-03-16T20:07:05.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b1ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b1ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40374/0 (socket says 172.21.15.32:40374) 2024-03-16T20:07:05.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b1ffff640 1 -- 172.21.15.32:0/2887298606 learned_addr learned my addr 172.21.15.32:0/2887298606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:05.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b1effd640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b2006f980 0x7f9b20149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.634+0000 7f9b1f7fe640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b2006f5a0 0x7f9b20157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 -- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b2006f980 msgr2=0x7f9b20149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b2006f980 0x7f9b20149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 -- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b2006f5a0 msgr2=0x7f9b20157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b2006f5a0 0x7f9b20157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 -- 172.21.15.32:0/2887298606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140094d0 con 0x7f9b200c8c40 2024-03-16T20:07:05.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.638+0000 7f9b1ffff640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c760 tx=0x7f9b1800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.642+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18006d60 con 0x7f9b200c8c40 2024-03-16T20:07:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.642+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18003980 con 0x7f9b200c8c40 2024-03-16T20:07:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.642+0000 7f9b25d8a640 1 -- 172.21.15.32:0/2887298606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2015d5f0 con 0x7f9b200c8c40 2024-03-16T20:07:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.642+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18003d30 con 0x7f9b200c8c40 2024-03-16T20:07:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.646+0000 7f9b25d8a640 1 -- 172.21.15.32:0/2887298606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b201f0060 con 0x7f9b200c8c40 2024-03-16T20:07:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.646+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af80052a0 con 0x7f9b200c8c40 2024-03-16T20:07:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.678+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9b180050d0 con 0x7f9b200c8c40 2024-03-16T20:07:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.678+0000 7f9b07fff640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 0x7f9b0c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.678+0000 7f9b1f7fe640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 0x7f9b0c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.678+0000 7f9b1f7fe640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 0x7f9b0c069330 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000aae0 tx=0x7f9b10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:05.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.682+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9b18055cb0 con 0x7f9b200c8c40 2024-03-16T20:07:05.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:05.946+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9b18055a80 con 0x7f9b200c8c40 2024-03-16T20:07:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.274+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9af8005900 con 0x7f9b200c8c40 2024-03-16T20:07:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.306+0000 7f9b07fff640 1 -- 172.21.15.32:0/2887298606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f9af8005900 con 0x7f9b200c8c40 2024-03-16T20:07:06.294 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:06.294 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 msgr2=0x7f9b0c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 0x7f9b0c069330 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000aae0 tx=0x7f9b10008040 comp rx=0 tx=0).stop 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 msgr2=0x7f9b2015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c760 tx=0x7f9b1800cb60 comp rx=0 tx=0).stop 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 shutdown_connections 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9b0c066e70 0x7f9b0c069330 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b200c8c40 0x7f9b2015d0b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9b2006f980 0x7f9b20149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 --2- 172.21.15.32:0/2887298606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b2006f5a0 0x7f9b20157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 >> 172.21.15.32:0/2887298606 conn(0x7f9b20066cd0 msgr2=0x7f9b20152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 shutdown_connections 2024-03-16T20:07:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:06.310+0000 7f9b05ffb640 1 -- 172.21.15.32:0/2887298606 wait complete. 2024-03-16T20:07:06.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:06.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:07.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:08.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 Processor -- start 2024-03-16T20:07:08.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 -- start start 2024-03-16T20:07:08.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc8f4073050 0x7fc8f4150c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f40c16a0 0x7fc8f4151160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f41516a0 con 0x7fc8f4072c70 2024-03-16T20:07:08.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f4156f20 con 0x7fc8f4073050 2024-03-16T20:07:08.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.054+0000 7fc8fb39d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f406e6a0 con 0x7fc8f40c16a0 2024-03-16T20:07:08.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.058+0000 7fc8f9139640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc8f4072c70 0x7fc8f40c0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44434/0 (socket says 172.21.15.32:44434) 2024-03-16T20:07:08.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.058+0000 7fc8f9139640 1 -- 172.21.15.32:0/2509184886 learned_addr learned my addr 172.21.15.32:0/2509184886 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:08.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.058+0000 7fc8f8938640 1 --2- 172.21.15.32:0/2509184886 >> v2:172.21.15.59:3300/0 conn(0x7fc8f4073050 0x7fc8f4150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.058+0000 7fc8f993a640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f40c16a0 0x7fc8f4151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8ebfff640 1 -- 172.21.15.32:0/2509184886 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963150667 0 0) 0x7fc8f41516a0 con 0x7fc8f4072c70 2024-03-16T20:07:08.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8ebfff640 1 -- 172.21.15.32:0/2509184886 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e0003530 con 0x7fc8f4072c70 2024-03-16T20:07:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8f8938640 1 -- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f40c16a0 msgr2=0x7fc8f4151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8f8938640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f40c16a0 0x7fc8f4151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8f8938640 1 -- 172.21.15.32:0/2509184886 >> v1:172.21.15.32:6789/0 conn(0x7fc8f4072c70 legacy=0x7fc8f40c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.062+0000 7fc8f8938640 1 -- 172.21.15.32:0/2509184886 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f406c250 con 0x7fc8f4073050 2024-03-16T20:07:08.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.066+0000 7fc8f8938640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4073050 0x7fc8f4150c20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40097c0 tx=0x7fc8e401c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4f50e7a154153a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8ebfff640 1 -- 172.21.15.32:0/2509184886 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40ab670 con 0x7fc8f4073050 2024-03-16T20:07:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8ebfff640 1 -- 172.21.15.32:0/2509184886 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e40ab7d0 con 0x7fc8f4073050 2024-03-16T20:07:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8ebfff640 1 -- 172.21.15.32:0/2509184886 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40abb60 con 0x7fc8f4073050 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 -- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4073050 msgr2=0x7fc8f4150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4073050 0x7fc8f4150c20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40097c0 tx=0x7fc8e401c490 comp rx=0 tx=0).stop 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 -- 172.21.15.32:0/2509184886 shutdown_connections 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f40c16a0 0x7fc8f4151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/2509184886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4073050 0x7fc8f4150c20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 -- 172.21.15.32:0/2509184886 >> 172.21.15.32:0/2509184886 conn(0x7fc8f406afc0 msgr2=0x7fc8f406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 -- 172.21.15.32:0/2509184886 shutdown_connections 2024-03-16T20:07:08.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.074+0000 7fc8fb39d640 1 -- 172.21.15.32:0/2509184886 wait complete. 2024-03-16T20:07:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 Processor -- start 2024-03-16T20:07:08.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 -- start start 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f4073050 0x7fc8f4141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8f40c16a0 0x7fc8f4141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4140b70 con 0x7fc8f40c16a0 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f414f500 con 0x7fc8f4072c70 2024-03-16T20:07:08.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8fb39d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f414e9a0 con 0x7fc8f4073050 2024-03-16T20:07:08.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8f9139640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8f993a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8f40c16a0 0x7fc8f4141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8f9139640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:49916/0 (socket says 172.21.15.32:49916) 2024-03-16T20:07:08.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8f9139640 1 -- 172.21.15.32:0/1306638282 learned_addr learned my addr 172.21.15.32:0/1306638282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:08.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.078+0000 7fc8f8938640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f4073050 0x7fc8f4141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.086+0000 7fc8f9139640 1 -- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f4073050 msgr2=0x7fc8f4141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.086+0000 7fc8f9139640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f4073050 0x7fc8f4141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.086+0000 7fc8f9139640 1 -- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8f40c16a0 msgr2=0x7fc8f4141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.086+0000 7fc8f9139640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8f40c16a0 0x7fc8f4141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.086+0000 7fc8f9139640 1 -- 172.21.15.32:0/1306638282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009420 con 0x7fc8f4072c70 2024-03-16T20:07:08.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.090+0000 7fc8f9139640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000e6e0 tx=0x7fc8f000eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:08.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.094+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0005ac0 con 0x7fc8f4072c70 2024-03-16T20:07:08.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.094+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f414ec60 con 0x7fc8f4072c70 2024-03-16T20:07:08.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.094+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f414f220 con 0x7fc8f4072c70 2024-03-16T20:07:08.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.098+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f4140940 con 0x7fc8f4072c70 2024-03-16T20:07:08.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.098+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0005c20 con 0x7fc8f4072c70 2024-03-16T20:07:08.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.098+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f000c850 con 0x7fc8f4072c70 2024-03-16T20:07:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.430+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc8f000c9b0 con 0x7fc8f4072c70 2024-03-16T20:07:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.430+0000 7fc8e9ffb640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 0x7fc8d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.430+0000 7fc8f8938640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 0x7fc8d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.430+0000 7fc8f8938640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 0x7fc8d8069300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40af000 tx=0x7fc8e40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:08.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.434+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc8f0057c40 con 0x7fc8f4072c70 2024-03-16T20:07:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.458+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc8f0053ee0 con 0x7fc8f4072c70 2024-03-16T20:07:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.702+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8f4140940 con 0x7fc8f4072c70 2024-03-16T20:07:08.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.738+0000 7fc8e9ffb640 1 -- 172.21.15.32:0/1306638282 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc8f4140940 con 0x7fc8f4072c70 2024-03-16T20:07:08.728 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:08.728 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:08.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 msgr2=0x7fc8d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 0x7fc8d8069300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40af000 tx=0x7fc8e40a7000 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 msgr2=0x7fc8f40c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000e6e0 tx=0x7fc8f000eba0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 shutdown_connections 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8d8066e40 0x7fc8d8069300 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8f40c16a0 0x7fc8f4141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8f4073050 0x7fc8f4141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 --2- 172.21.15.32:0/1306638282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f4072c70 0x7fc8f40c4a70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 >> 172.21.15.32:0/1306638282 conn(0x7fc8f406afc0 msgr2=0x7fc8f4157900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 shutdown_connections 2024-03-16T20:07:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:08.742+0000 7fc8fb39d640 1 -- 172.21.15.32:0/1306638282 wait complete. 2024-03-16T20:07:09.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:09.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:10.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:10.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 Processor -- start 2024-03-16T20:07:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 -- start start 2024-03-16T20:07:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa4001658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4001546a0 con 0x7fa400070360 2024-03-16T20:07:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa400159f20 con 0x7fa4001542c0 2024-03-16T20:07:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.470+0000 7fa406cab640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015a1e0 con 0x7fa400073010 2024-03-16T20:07:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.474+0000 7fa3fffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa400070360 0x7fa40015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52498/0 (socket says 172.21.15.32:52498) 2024-03-16T20:07:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.474+0000 7fa3fffff640 1 -- 172.21.15.32:0/3500577309 learned_addr learned my addr 172.21.15.32:0/3500577309 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.474+0000 7fa405248640 1 --2- 172.21.15.32:0/3500577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa4001658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:10.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.478+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/3500577309 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407101723 0 0) 0x7fa4001546a0 con 0x7fa400070360 2024-03-16T20:07:10.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.478+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/3500577309 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f0003530 con 0x7fa400070360 2024-03-16T20:07:10.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.478+0000 7fa405248640 1 -- 172.21.15.32:0/3500577309 >> v1:172.21.15.59:6789/0 conn(0x7fa4001542c0 legacy=0x7fa400072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:10.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.478+0000 7fa405248640 1 -- 172.21.15.32:0/3500577309 >> v1:172.21.15.32:6789/0 conn(0x7fa400070360 legacy=0x7fa40015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:10.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.478+0000 7fa405248640 1 -- 172.21.15.32:0/3500577309 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40015a4c0 con 0x7fa400073010 2024-03-16T20:07:10.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.482+0000 7fa405248640 1 --2- 172.21.15.32:0/3500577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa4001658b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009dc0 tx=0x7fa3f801c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e6a6d2bc0c453d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:10.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/3500577309 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80a5940 con 0x7fa400073010 2024-03-16T20:07:10.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/3500577309 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f80a5aa0 con 0x7fa400073010 2024-03-16T20:07:10.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa3ff7fe640 1 -- 172.21.15.32:0/3500577309 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80a5e30 con 0x7fa400073010 2024-03-16T20:07:10.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 -- 172.21.15.32:0/3500577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 msgr2=0x7fa4001658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 --2- 172.21.15.32:0/3500577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa4001658b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009dc0 tx=0x7fa3f801c990 comp rx=0 tx=0).stop 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 -- 172.21.15.32:0/3500577309 shutdown_connections 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 --2- 172.21.15.32:0/3500577309 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa4001658b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 -- 172.21.15.32:0/3500577309 >> 172.21.15.32:0/3500577309 conn(0x7fa40006c0d0 msgr2=0x7fa40006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 -- 172.21.15.32:0/3500577309 shutdown_connections 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 -- 172.21.15.32:0/3500577309 wait complete. 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.490+0000 7fa406cab640 1 Processor -- start 2024-03-16T20:07:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 -- start start 2024-03-16T20:07:10.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:10.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa400141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:10.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4001542c0 0x7fa40013cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:10.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa400148a90 con 0x7fa400070360 2024-03-16T20:07:10.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015b7d0 con 0x7fa4001542c0 2024-03-16T20:07:10.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa406cab640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015b210 con 0x7fa400073010 2024-03-16T20:07:10.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa404a47640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:10.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa3fffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa400141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:10.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa405248640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4001542c0 0x7fa40013cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:10.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa404a47640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49806/0 (socket says 172.21.15.32:49806) 2024-03-16T20:07:10.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.494+0000 7fa404a47640 1 -- 172.21.15.32:0/4084375293 learned_addr learned my addr 172.21.15.32:0/4084375293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:10.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.498+0000 7fa404a47640 1 -- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 msgr2=0x7fa400141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:10.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.498+0000 7fa404a47640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa400141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:10.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.498+0000 7fa404a47640 1 -- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4001542c0 msgr2=0x7fa40013cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:10.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.498+0000 7fa404a47640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4001542c0 0x7fa40013cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:10.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.498+0000 7fa404a47640 1 -- 172.21.15.32:0/4084375293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8009a70 con 0x7fa400070360 2024-03-16T20:07:10.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.502+0000 7fa404a47640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c3c0 tx=0x7fa3f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.506+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40052d0 con 0x7fa400070360 2024-03-16T20:07:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.506+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4005430 con 0x7fa400070360 2024-03-16T20:07:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.506+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003840 con 0x7fa400070360 2024-03-16T20:07:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.506+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40015b4f0 con 0x7fa400070360 2024-03-16T20:07:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.506+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4000c65b0 con 0x7fa400070360 2024-03-16T20:07:10.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.510+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400070b00 con 0x7fa400070360 2024-03-16T20:07:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.538+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa3f40039a0 con 0x7fa400070360 2024-03-16T20:07:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.538+0000 7fa3fd7fa640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 0x7fa3e4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.542+0000 7fa3fffff640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 0x7fa3e4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.542+0000 7fa3fffff640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 0x7fa3e4069380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa400142ea0 tx=0x7fa3ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.546+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa3f4054f20 con 0x7fa400070360 2024-03-16T20:07:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:10.890+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa3f40511c0 con 0x7fa400070360 2024-03-16T20:07:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.110+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa400068000 con 0x7fa400070360 2024-03-16T20:07:11.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.134+0000 7fa3fd7fa640 1 -- 172.21.15.32:0/4084375293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa3f4054880 con 0x7fa400070360 2024-03-16T20:07:11.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:11.123 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:11.123 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 msgr2=0x7fa3e4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 0x7fa3e4069380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa400142ea0 tx=0x7fa3ec009040 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 msgr2=0x7fa4001415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c3c0 tx=0x7fa3f400c880 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 shutdown_connections 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa3e4066ec0 0x7fa3e4069380 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4001542c0 0x7fa40013cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa400073010 0x7fa400141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 --2- 172.21.15.32:0/4084375293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa400070360 0x7fa4001415a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 >> 172.21.15.32:0/4084375293 conn(0x7fa40006c0d0 msgr2=0x7fa400158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 shutdown_connections 2024-03-16T20:07:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:11.138+0000 7fa406cab640 1 -- 172.21.15.32:0/4084375293 wait complete. 2024-03-16T20:07:11.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:11.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:12.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:07:12.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:07:12.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 Processor -- start 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 -- start start 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97a80c9de0 0x7f97a8072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 0x7f97a80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a806b3f0 con 0x7f97a8154030 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a806b990 con 0x7f97a80c9de0 2024-03-16T20:07:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.886+0000 7f97af622640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a806b6b0 con 0x7f97a8154410 2024-03-16T20:07:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.890+0000 7f97acbbd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97a80c9de0 0x7f97a8072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.890+0000 7f97adbbf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 0x7f97a80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:12.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.890+0000 7f97adbbf640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 0x7f97a80cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:47046/0 (socket says 172.21.15.32:47046) 2024-03-16T20:07:12.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.890+0000 7f97adbbf640 1 -- 172.21.15.32:0/3317152201 learned_addr learned my addr 172.21.15.32:0/3317152201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.894+0000 7f97acbbd640 1 -- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 msgr2=0x7f97a80cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.894+0000 7f97acbbd640 1 --2- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 0x7f97a80cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.894+0000 7f97acbbd640 1 -- 172.21.15.32:0/3317152201 >> v1:172.21.15.59:6789/0 conn(0x7f97a8154030 legacy=0x7f97a8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.894+0000 7f97acbbd640 1 -- 172.21.15.32:0/3317152201 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80c9300 con 0x7f97a80c9de0 2024-03-16T20:07:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.902+0000 7f97acbbd640 1 --2- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80c9de0 0x7f97a8072ec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97980097c0 tx=0x7f979801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c2cf1d3a4ebea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f979ffff640 1 -- 172.21.15.32:0/3317152201 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97980a8990 con 0x7f97a80c9de0 2024-03-16T20:07:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f979ffff640 1 -- 172.21.15.32:0/3317152201 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97980a8af0 con 0x7f97a80c9de0 2024-03-16T20:07:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f979ffff640 1 -- 172.21.15.32:0/3317152201 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97980a8e80 con 0x7f97a80c9de0 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80c9de0 msgr2=0x7f97a8072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80c9de0 0x7f97a8072ec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97980097c0 tx=0x7f979801ec90 comp rx=0 tx=0).stop 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- 172.21.15.32:0/3317152201 shutdown_connections 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154410 0x7f97a80cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- 172.21.15.32:0/3317152201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80c9de0 0x7f97a8072ec0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- 172.21.15.32:0/3317152201 >> 172.21.15.32:0/3317152201 conn(0x7f97a806e6c0 msgr2=0x7f97a806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- 172.21.15.32:0/3317152201 shutdown_connections 2024-03-16T20:07:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- 172.21.15.32:0/3317152201 wait complete. 2024-03-16T20:07:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 Processor -- start 2024-03-16T20:07:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- start start 2024-03-16T20:07:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154030 0x7f97a80cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a8154410 0x7f97a80cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8157150 con 0x7f97a80c9de0 2024-03-16T20:07:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80ca860 con 0x7f97a8154410 2024-03-16T20:07:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.914+0000 7f97af622640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81568d0 con 0x7f97a8154030 2024-03-16T20:07:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.918+0000 7f97ad3be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.918+0000 7f97acbbd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154030 0x7f97a80cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.918+0000 7f97ad3be640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49812/0 (socket says 172.21.15.32:49812) 2024-03-16T20:07:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.918+0000 7f97ad3be640 1 -- 172.21.15.32:0/1015965957 learned_addr learned my addr 172.21.15.32:0/1015965957 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.918+0000 7f97adbbf640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a8154410 0x7f97a80cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.922+0000 7f97ad3be640 1 -- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154030 msgr2=0x7f97a80cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.922+0000 7f97ad3be640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154030 0x7f97a80cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.922+0000 7f97ad3be640 1 -- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a8154410 msgr2=0x7f97a80cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.922+0000 7f97ad3be640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a8154410 0x7f97a80cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.922+0000 7f97ad3be640 1 -- 172.21.15.32:0/1015965957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9798009420 con 0x7f97a80c9de0 2024-03-16T20:07:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.926+0000 7f97ad3be640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c420 tx=0x7f97a400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.930+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40052d0 con 0x7f97a80c9de0 2024-03-16T20:07:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.930+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a4005430 con 0x7f97a80c9de0 2024-03-16T20:07:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.930+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81573a0 con 0x7f97a80c9de0 2024-03-16T20:07:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.930+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4007260 con 0x7f97a80c9de0 2024-03-16T20:07:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.930+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8144cf0 con 0x7f97a80c9de0 2024-03-16T20:07:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:12.934+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a8068000 con 0x7f97a80c9de0 2024-03-16T20:07:13.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.198+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f97a40073c0 con 0x7f97a80c9de0 2024-03-16T20:07:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.198+0000 7f979dffb640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 0x7f978c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.202+0000 7f97acbbd640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 0x7f978c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.202+0000 7f97acbbd640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 0x7f978c069350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f97980b2000 tx=0x7f97980aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.226+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f97a4007670 con 0x7f97a80c9de0 2024-03-16T20:07:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.226+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f97a40514d0 con 0x7f97a80c9de0 2024-03-16T20:07:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.578+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97a80c8bb0 con 0x7f97a80c9de0 2024-03-16T20:07:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f979dffb640 1 -- 172.21.15.32:0/1015965957 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f97a4020020 con 0x7f97a80c9de0 2024-03-16T20:07:13.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:13.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 msgr2=0x7f978c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 0x7f978c069350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f97980b2000 tx=0x7f97980aa070 comp rx=0 tx=0).stop 2024-03-16T20:07:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 msgr2=0x7f97a8155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c420 tx=0x7f97a400c8e0 comp rx=0 tx=0).stop 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 shutdown_connections 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f978c066e90 0x7f978c069350 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a8154410 0x7f97a80cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f97a8154030 0x7f97a80cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 --2- 172.21.15.32:0/1015965957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a80c9de0 0x7f97a8155570 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 >> 172.21.15.32:0/1015965957 conn(0x7f97a806e6c0 msgr2=0x7f97a8151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 shutdown_connections 2024-03-16T20:07:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:13.610+0000 7f97af622640 1 -- 172.21.15.32:0/1015965957 wait complete. 2024-03-16T20:07:14.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:14.220 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:15.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:15.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 Processor -- start 2024-03-16T20:07:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 -- start start 2024-03-16T20:07:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 0x7fba9015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fba900c3570 0x7fba900c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9006c460 con 0x7fba9015ac10 2024-03-16T20:07:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba900c3f40 con 0x7fba901540c0 2024-03-16T20:07:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba9856a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9015a240 con 0x7fba900c3570 2024-03-16T20:07:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba96b07640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fba900c3570 0x7fba900c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba96306640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 0x7fba9015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba96306640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 0x7fba9015a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49820/0 (socket says 172.21.15.32:49820) 2024-03-16T20:07:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.338+0000 7fba96306640 1 -- 172.21.15.32:0/3957757692 learned_addr learned my addr 172.21.15.32:0/3957757692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.342+0000 7fba96b07640 1 -- 172.21.15.32:0/3957757692 >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 msgr2=0x7fba9015a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.342+0000 7fba96b07640 1 --2- 172.21.15.32:0/3957757692 >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 0x7fba9015a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.342+0000 7fba96b07640 1 -- 172.21.15.32:0/3957757692 >> v1:172.21.15.32:6789/0 conn(0x7fba9015ac10 legacy=0x7fba900c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.342+0000 7fba96b07640 1 -- 172.21.15.32:0/3957757692 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9006cf20 con 0x7fba900c3570 2024-03-16T20:07:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.350+0000 7fba96b07640 1 --2- 172.21.15.32:0/3957757692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba900c3a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fba9006c740 tx=0x7fba8801c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796c2d49f473ba2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba95304640 1 -- 172.21.15.32:0/3957757692 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba880a59c0 con 0x7fba900c3570 2024-03-16T20:07:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba95304640 1 -- 172.21.15.32:0/3957757692 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba880a5b20 con 0x7fba900c3570 2024-03-16T20:07:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba95304640 1 -- 172.21.15.32:0/3957757692 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba880ad440 con 0x7fba900c3570 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba9856a640 1 -- 172.21.15.32:0/3957757692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 msgr2=0x7fba900c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba9856a640 1 --2- 172.21.15.32:0/3957757692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba900c3a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fba9006c740 tx=0x7fba8801c900 comp rx=0 tx=0).stop 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba9856a640 1 -- 172.21.15.32:0/3957757692 shutdown_connections 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba9856a640 1 --2- 172.21.15.32:0/3957757692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba900c3a00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.354+0000 7fba9856a640 1 --2- 172.21.15.32:0/3957757692 >> v2:172.21.15.32:3300/0 conn(0x7fba901540c0 0x7fba9015a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- 172.21.15.32:0/3957757692 >> 172.21.15.32:0/3957757692 conn(0x7fba90070f90 msgr2=0x7fba900713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- 172.21.15.32:0/3957757692 shutdown_connections 2024-03-16T20:07:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- 172.21.15.32:0/3957757692 wait complete. 2024-03-16T20:07:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 Processor -- start 2024-03-16T20:07:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- start start 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba90152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba9015ac10 0x7fba9014b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90153380 con 0x7fba901540c0 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90146ae0 con 0x7fba900c3570 2024-03-16T20:07:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba9856a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba90159470 con 0x7fba9015ac10 2024-03-16T20:07:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.358+0000 7fba95b05640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.362+0000 7fba96b07640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba9015ac10 0x7fba9014b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.362+0000 7fba96306640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba90152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.362+0000 7fba95b05640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49824/0 (socket says 172.21.15.32:49824) 2024-03-16T20:07:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.362+0000 7fba95b05640 1 -- 172.21.15.32:0/1598358829 learned_addr learned my addr 172.21.15.32:0/1598358829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 -- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba9015ac10 msgr2=0x7fba9014b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba9015ac10 0x7fba9014b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 -- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 msgr2=0x7fba90152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba90152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 -- 172.21.15.32:0/1598358829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88009a70 con 0x7fba901540c0 2024-03-16T20:07:15.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.366+0000 7fba95b05640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fba8000c370 tx=0x7fba8000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:15.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.370+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba800052d0 con 0x7fba901540c0 2024-03-16T20:07:15.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.370+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba80005430 con 0x7fba901540c0 2024-03-16T20:07:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.370+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba90159750 con 0x7fba901540c0 2024-03-16T20:07:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.370+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba80007260 con 0x7fba901540c0 2024-03-16T20:07:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.370+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90154810 con 0x7fba901540c0 2024-03-16T20:07:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.374+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90070390 con 0x7fba901540c0 2024-03-16T20:07:15.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.670+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fba800073c0 con 0x7fba901540c0 2024-03-16T20:07:15.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.670+0000 7fba86ffd640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 0x7fba74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:15.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.670+0000 7fba96306640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 0x7fba74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:15.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.670+0000 7fba96306640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 0x7fba74069300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009b50 tx=0x7fba8c009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.698+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fba80007670 con 0x7fba901540c0 2024-03-16T20:07:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.698+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fba80051110 con 0x7fba901540c0 2024-03-16T20:07:15.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.902+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba90068000 con 0x7fba901540c0 2024-03-16T20:07:15.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.962+0000 7fba86ffd640 1 -- 172.21.15.32:0/1598358829 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fba800547d0 con 0x7fba901540c0 2024-03-16T20:07:15.950 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:15.951 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:15.951 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 msgr2=0x7fba74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 0x7fba74069300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009b50 tx=0x7fba8c009240 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 msgr2=0x7fba901465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fba8000c370 tx=0x7fba8000c830 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 shutdown_connections 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fba74066e40 0x7fba74069300 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fba9015ac10 0x7fba9014b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba901540c0 0x7fba901465a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 --2- 172.21.15.32:0/1598358829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba900c3570 0x7fba90152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 >> 172.21.15.32:0/1598358829 conn(0x7fba90070f90 msgr2=0x7fba900c91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 shutdown_connections 2024-03-16T20:07:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:15.966+0000 7fba9856a640 1 -- 172.21.15.32:0/1598358829 wait complete. 2024-03-16T20:07:16.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:16.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:17.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 Processor -- start 2024-03-16T20:07:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 -- start start 2024-03-16T20:07:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3804151000 0x7f38040c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c8980 0x7f380415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38040c8ec0 con 0x7f38040c36b0 2024-03-16T20:07:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3804072f10 con 0x7f3804151000 2024-03-16T20:07:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3808f9f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38040c5000 con 0x7f38040c8980 2024-03-16T20:07:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f380259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3804151000 0x7f38040c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.702+0000 7f3802d9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c8980 0x7f380415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.706+0000 7f380259c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3804151000 0x7f38040c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49826/0 (socket says 172.21.15.32:49826) 2024-03-16T20:07:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.706+0000 7f380259c640 1 -- 172.21.15.32:0/1970274684 learned_addr learned my addr 172.21.15.32:0/1970274684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.710+0000 7f380259c640 1 -- 172.21.15.32:0/1970274684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c8980 msgr2=0x7f380415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.710+0000 7f380259c640 1 --2- 172.21.15.32:0/1970274684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c8980 0x7f380415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.710+0000 7f380259c640 1 -- 172.21.15.32:0/1970274684 >> v1:172.21.15.32:6789/0 conn(0x7f38040c36b0 legacy=0x7f38040c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.710+0000 7f380259c640 1 -- 172.21.15.32:0/1970274684 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804072950 con 0x7f3804151000 2024-03-16T20:07:17.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.714+0000 7f380259c640 1 --2- 172.21.15.32:0/1970274684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804151000 0x7f38040c8420 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009870 tx=0x7f37f401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd0cb94ac8a8478 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f380159a640 1 -- 172.21.15.32:0/1970274684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f40a9620 con 0x7f3804151000 2024-03-16T20:07:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f380159a640 1 -- 172.21.15.32:0/1970274684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f40a9780 con 0x7f3804151000 2024-03-16T20:07:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f380159a640 1 -- 172.21.15.32:0/1970274684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f40a9b10 con 0x7f3804151000 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 -- 172.21.15.32:0/1970274684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804151000 msgr2=0x7f38040c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1970274684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804151000 0x7f38040c8420 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009870 tx=0x7f37f401c560 comp rx=0 tx=0).stop 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 -- 172.21.15.32:0/1970274684 shutdown_connections 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1970274684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c8980 0x7f380415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1970274684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804151000 0x7f38040c8420 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 -- 172.21.15.32:0/1970274684 >> 172.21.15.32:0/1970274684 conn(0x7f380406c0d0 msgr2=0x7f380406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 -- 172.21.15.32:0/1970274684 shutdown_connections 2024-03-16T20:07:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.722+0000 7f3808f9f640 1 -- 172.21.15.32:0/1970274684 wait complete. 2024-03-16T20:07:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 Processor -- start 2024-03-16T20:07:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 -- start start 2024-03-16T20:07:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c36b0 0x7f3804147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3804151000 0x7f380413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3804147860 con 0x7f38040c8980 2024-03-16T20:07:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38041f0a30 con 0x7f3804151000 2024-03-16T20:07:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3808f9f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3804159b70 con 0x7f38040c36b0 2024-03-16T20:07:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3801d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f380259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c36b0 0x7f3804147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3801d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49834/0 (socket says 172.21.15.32:49834) 2024-03-16T20:07:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3801d9b640 1 -- 172.21.15.32:0/1944174151 learned_addr learned my addr 172.21.15.32:0/1944174151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.726+0000 7f3802d9d640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3804151000 0x7f380413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.730+0000 7f3801d9b640 1 -- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c36b0 msgr2=0x7f3804147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.730+0000 7f3801d9b640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c36b0 0x7f3804147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.730+0000 7f3801d9b640 1 -- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3804151000 msgr2=0x7f380413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.730+0000 7f3801d9b640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3804151000 0x7f380413f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.730+0000 7f3801d9b640 1 -- 172.21.15.32:0/1944174151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40094d0 con 0x7f38040c8980 2024-03-16T20:07:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.734+0000 7f3801d9b640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c430 tx=0x7f37f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.738+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80052d0 con 0x7f38040c8980 2024-03-16T20:07:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.738+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f8005430 con 0x7f38040c8980 2024-03-16T20:07:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.738+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3804159e50 con 0x7f38040c8980 2024-03-16T20:07:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.738+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8007260 con 0x7f38040c8980 2024-03-16T20:07:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.738+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38041f0fd0 con 0x7f38040c8980 2024-03-16T20:07:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.742+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38040c90e0 con 0x7f38040c8980 2024-03-16T20:07:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.770+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f37f80073c0 con 0x7f38040c8980 2024-03-16T20:07:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.770+0000 7f37f2ffd640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 0x7f37e4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.770+0000 7f380259c640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 0x7f37e4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:17.774+0000 7f380259c640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 0x7f37e4069330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37f40ae000 tx=0x7f37f40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:18.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.054+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f37f8055210 con 0x7f38040c8980 2024-03-16T20:07:18.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.054+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f37f80514b0 con 0x7f38040c8980 2024-03-16T20:07:18.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.258+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3804068000 con 0x7f38040c8980 2024-03-16T20:07:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.282+0000 7f37f2ffd640 1 -- 172.21.15.32:0/1944174151 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f37f8020070 con 0x7f38040c8980 2024-03-16T20:07:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:18.272 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 msgr2=0x7f37e4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 0x7f37e4069330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37f40ae000 tx=0x7f37f40a7000 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 msgr2=0x7f380414c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c430 tx=0x7f37f800c8f0 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 shutdown_connections 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f37e4066e70 0x7f37e4069330 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3804151000 0x7f380413f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040c8980 0x7f380414c530 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 --2- 172.21.15.32:0/1944174151 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38040c36b0 0x7f3804147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 >> 172.21.15.32:0/1944174151 conn(0x7f380406c0d0 msgr2=0x7f380414eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 shutdown_connections 2024-03-16T20:07:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:18.286+0000 7f3808f9f640 1 -- 172.21.15.32:0/1944174151 wait complete. 2024-03-16T20:07:18.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:18.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:19.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:20.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 Processor -- start 2024-03-16T20:07:20.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 -- start start 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f262c0c8550 0x7f262c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262c15ab60 con 0x7f262c1540c0 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262c159fb0 con 0x7f262c1544a0 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f2631ff2640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262c06fb80 con 0x7f262c0c8550 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f262bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f262c0c8550 0x7f262c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f262b7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f262c1540c0 0x7f262c06f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49678/0 (socket says 172.21.15.32:49678) 2024-03-16T20:07:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.018+0000 7f262b7fe640 1 -- 172.21.15.32:0/2916717998 learned_addr learned my addr 172.21.15.32:0/2916717998 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:20.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.022+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404869202 0 0) 0x7f262c15ab60 con 0x7f262c1540c0 2024-03-16T20:07:20.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.022+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2614003530 con 0x7f262c1540c0 2024-03-16T20:07:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.026+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661997312 0 0) 0x7f262c159fb0 con 0x7f262c1544a0 2024-03-16T20:07:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.026+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c15ab60 con 0x7f262c1544a0 2024-03-16T20:07:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.026+0000 7f262bfff640 1 -- 172.21.15.32:0/2916717998 >> v1:172.21.15.59:6789/0 conn(0x7f262c1544a0 legacy=0x7f262c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.026+0000 7f262bfff640 1 -- 172.21.15.32:0/2916717998 >> v1:172.21.15.32:6789/0 conn(0x7f262c1540c0 legacy=0x7f262c06f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.026+0000 7f262bfff640 1 -- 172.21.15.32:0/2916717998 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c15a230 con 0x7f262c0c8550 2024-03-16T20:07:20.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.034+0000 7f262bfff640 1 --2- 172.21.15.32:0/2916717998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c0c8550 0x7f262c0cd830 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2624009d90 tx=0x7f262401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c720bafc4f285d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:20.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240a5810 con 0x7f262c0c8550 2024-03-16T20:07:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26240a5970 con 0x7f262c0c8550 2024-03-16T20:07:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f262a7fc640 1 -- 172.21.15.32:0/2916717998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240a5d00 con 0x7f262c0c8550 2024-03-16T20:07:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- 172.21.15.32:0/2916717998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c0c8550 msgr2=0x7f262c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 --2- 172.21.15.32:0/2916717998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c0c8550 0x7f262c0cd830 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2624009d90 tx=0x7f262401c970 comp rx=0 tx=0).stop 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- 172.21.15.32:0/2916717998 shutdown_connections 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 --2- 172.21.15.32:0/2916717998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c0c8550 0x7f262c0cd830 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- 172.21.15.32:0/2916717998 >> 172.21.15.32:0/2916717998 conn(0x7f262c066ea0 msgr2=0x7f262c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- 172.21.15.32:0/2916717998 shutdown_connections 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- 172.21.15.32:0/2916717998 wait complete. 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 Processor -- start 2024-03-16T20:07:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.042+0000 7f2631ff2640 1 -- start start 2024-03-16T20:07:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f262c0c8550 0x7f262c14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c1544a0 0x7f262c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c0c0ef0 con 0x7f262c1544a0 2024-03-16T20:07:20.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c14cae0 con 0x7f262c1540c0 2024-03-16T20:07:20.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f2631ff2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262c14cdc0 con 0x7f262c0c8550 2024-03-16T20:07:20.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f262affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:20.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f262b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f262c0c8550 0x7f262c14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:20.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f262affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:59566/0 (socket says 172.21.15.32:59566) 2024-03-16T20:07:20.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.046+0000 7f262affd640 1 -- 172.21.15.32:0/685441863 learned_addr learned my addr 172.21.15.32:0/685441863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.058+0000 7f262affd640 1 -- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f262c0c8550 msgr2=0x7f262c14af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.058+0000 7f262affd640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f262c0c8550 0x7f262c14af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.058+0000 7f262affd640 1 -- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c1544a0 msgr2=0x7f262c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.058+0000 7f262affd640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c1544a0 0x7f262c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.058+0000 7f262affd640 1 -- 172.21.15.32:0/685441863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624009a70 con 0x7f262c1540c0 2024-03-16T20:07:20.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.062+0000 7f262affd640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f262000c430 tx=0x7f262000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26200052d0 con 0x7f262c1540c0 2024-03-16T20:07:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2620005450 con 0x7f262c1540c0 2024-03-16T20:07:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2620003840 con 0x7f262c1540c0 2024-03-16T20:07:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c14d0a0 con 0x7f262c1540c0 2024-03-16T20:07:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c0c7dc0 con 0x7f262c1540c0 2024-03-16T20:07:20.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.070+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c14c560 con 0x7f262c1540c0 2024-03-16T20:07:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.390+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f26200039a0 con 0x7f262c1540c0 2024-03-16T20:07:20.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.390+0000 7f2613fff640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 0x7f2618069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:20.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.390+0000 7f262b7fe640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 0x7f2618069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:20.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.390+0000 7f262b7fe640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 0x7f2618069350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f261c000f70 tx=0x7f261c007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.398+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2620054ed0 con 0x7f262c1540c0 2024-03-16T20:07:20.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.430+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2620051170 con 0x7f262c1540c0 2024-03-16T20:07:20.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.646+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f262c14c560 con 0x7f262c1540c0 2024-03-16T20:07:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.666+0000 7f2613fff640 1 -- 172.21.15.32:0/685441863 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f262c14c560 con 0x7f262c1540c0 2024-03-16T20:07:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:20.656 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 msgr2=0x7f2618069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 0x7f2618069350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f261c000f70 tx=0x7f261c007d00 comp rx=0 tx=0).stop 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 msgr2=0x7f262c0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f262000c430 tx=0x7f262000c8f0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 shutdown_connections 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2618066e90 0x7f2618069350 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c1544a0 0x7f262c1581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f262c1540c0 0x7f262c0cc620 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 --2- 172.21.15.32:0/685441863 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f262c0c8550 0x7f262c14af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 >> 172.21.15.32:0/685441863 conn(0x7f262c066ea0 msgr2=0x7f262c152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 shutdown_connections 2024-03-16T20:07:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:20.670+0000 7f2631ff2640 1 -- 172.21.15.32:0/685441863 wait complete. 2024-03-16T20:07:21.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:21.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:22.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:22.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 Processor -- start 2024-03-16T20:07:22.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 -- start start 2024-03-16T20:07:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0d000c3570 0x7f0d000c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0006c460 con 0x7f0d0015ac10 2024-03-16T20:07:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d000c3f60 con 0x7f0d000c3570 2024-03-16T20:07:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0d0600d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0015a240 con 0x7f0d001540c0 2024-03-16T20:07:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.402+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.406+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:47292/0 (socket says 172.21.15.32:47292) 2024-03-16T20:07:22.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.406+0000 7f0cff7fe640 1 -- 172.21.15.32:0/2482115053 learned_addr learned my addr 172.21.15.32:0/2482115053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:22.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.406+0000 7f0cfffff640 1 --2- 172.21.15.32:0/2482115053 >> v2:172.21.15.59:3300/0 conn(0x7f0d000c3570 0x7f0d000c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.410+0000 7f0cff7fe640 1 -- 172.21.15.32:0/2482115053 >> v2:172.21.15.59:3300/0 conn(0x7f0d000c3570 msgr2=0x7f0d000c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.410+0000 7f0cff7fe640 1 --2- 172.21.15.32:0/2482115053 >> v2:172.21.15.59:3300/0 conn(0x7f0d000c3570 0x7f0d000c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.410+0000 7f0cff7fe640 1 -- 172.21.15.32:0/2482115053 >> v1:172.21.15.59:6789/0 conn(0x7f0d0015ac10 legacy=0x7f0d000c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.410+0000 7f0cff7fe640 1 -- 172.21.15.32:0/2482115053 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0006cf20 con 0x7f0d001540c0 2024-03-16T20:07:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.418+0000 7f0cff7fe640 1 --2- 172.21.15.32:0/2482115053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009870 tx=0x7f0cf001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=594147a9b48c174b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:22.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0cfe7fc640 1 -- 172.21.15.32:0/2482115053 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00a95f0 con 0x7f0d001540c0 2024-03-16T20:07:22.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0cfe7fc640 1 -- 172.21.15.32:0/2482115053 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf00a9750 con 0x7f0d001540c0 2024-03-16T20:07:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0cfe7fc640 1 -- 172.21.15.32:0/2482115053 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00a9ae0 con 0x7f0d001540c0 2024-03-16T20:07:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 -- 172.21.15.32:0/2482115053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 msgr2=0x7f0d0015a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 --2- 172.21.15.32:0/2482115053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009870 tx=0x7f0cf001c530 comp rx=0 tx=0).stop 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 -- 172.21.15.32:0/2482115053 shutdown_connections 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 --2- 172.21.15.32:0/2482115053 >> v2:172.21.15.59:3300/0 conn(0x7f0d000c3570 0x7f0d000c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 --2- 172.21.15.32:0/2482115053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d0015a6d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 -- 172.21.15.32:0/2482115053 >> 172.21.15.32:0/2482115053 conn(0x7f0d00070f90 msgr2=0x7f0d000713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 -- 172.21.15.32:0/2482115053 shutdown_connections 2024-03-16T20:07:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.426+0000 7f0d0600d640 1 -- 172.21.15.32:0/2482115053 wait complete. 2024-03-16T20:07:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 Processor -- start 2024-03-16T20:07:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 -- start start 2024-03-16T20:07:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d000c3570 0x7f0d0014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d001f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d001f5bf0 con 0x7f0d0015ac10 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d001f28f0 con 0x7f0d000c3570 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0d0600d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d001f2bd0 con 0x7f0d001540c0 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0cfffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0cfffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41802/0 (socket says 172.21.15.32:41802) 2024-03-16T20:07:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0cfffff640 1 -- 172.21.15.32:0/76339676 learned_addr learned my addr 172.21.15.32:0/76339676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0cff7fe640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d000c3570 0x7f0d0014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.430+0000 7f0cfeffd640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d001f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.434+0000 7f0cfffff640 1 -- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 msgr2=0x7f0d001f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.434+0000 7f0cfffff640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d001f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.434+0000 7f0cfffff640 1 -- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d000c3570 msgr2=0x7f0d0014bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.434+0000 7f0cfffff640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d000c3570 0x7f0d0014bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.434+0000 7f0cfffff640 1 -- 172.21.15.32:0/76339676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00094d0 con 0x7f0d0015ac10 2024-03-16T20:07:22.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.438+0000 7f0cfffff640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8007e60 tx=0x7f0cf800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8014420 con 0x7f0d0015ac10 2024-03-16T20:07:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf80145a0 con 0x7f0d0015ac10 2024-03-16T20:07:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8014950 con 0x7f0d0015ac10 2024-03-16T20:07:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d001f2eb0 con 0x7f0d0015ac10 2024-03-16T20:07:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d000cb5e0 con 0x7f0d0015ac10 2024-03-16T20:07:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.446+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d00070390 con 0x7f0d0015ac10 2024-03-16T20:07:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.738+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0cf8014ab0 con 0x7f0d0015ac10 2024-03-16T20:07:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.738+0000 7f0ce7fff640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 0x7f0cec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.738+0000 7f0cff7fe640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 0x7f0cec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.738+0000 7f0cff7fe640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 0x7f0cec069350 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00ae000 tx=0x7f0cf00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.758+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0cf8056d80 con 0x7f0d0015ac10 2024-03-16T20:07:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:22.758+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0cf8053020 con 0x7f0d0015ac10 2024-03-16T20:07:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.117+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d00068000 con 0x7f0d0015ac10 2024-03-16T20:07:23.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.129+0000 7f0ce7fff640 1 -- 172.21.15.32:0/76339676 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f0cf80566e0 con 0x7f0d0015ac10 2024-03-16T20:07:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:23.121 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:23.121 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 msgr2=0x7f0cec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 0x7f0cec069350 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00ae000 tx=0x7f0cf00a7000 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 msgr2=0x7f0d001f65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8007e60 tx=0x7f0cf800e4b0 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 shutdown_connections 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0cec066e90 0x7f0cec069350 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d0015ac10 0x7f0d001f65e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0d001540c0 0x7f0d001f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 --2- 172.21.15.32:0/76339676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d000c3570 0x7f0d0014bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 >> 172.21.15.32:0/76339676 conn(0x7f0d00070f90 msgr2=0x7f0d000c7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:23.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 shutdown_connections 2024-03-16T20:07:23.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:23.133+0000 7f0d0600d640 1 -- 172.21.15.32:0/76339676 wait complete. 2024-03-16T20:07:23.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:23.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:24.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 Processor -- start 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 -- start start 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a0c15a420 0x7f5a0c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c072dd0 con 0x7f5a0c0725e0 2024-03-16T20:07:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c15d790 con 0x7f5a0c15a420 2024-03-16T20:07:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a11093640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c15ad90 con 0x7f5a0c150d70 2024-03-16T20:07:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a0b59e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a0c15a420 0x7f5a0c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a0b59e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a0c15a420 0x7f5a0c15a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41804/0 (socket says 172.21.15.32:41804) 2024-03-16T20:07:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.873+0000 7f5a0b59e640 1 -- 172.21.15.32:0/2999860454 learned_addr learned my addr 172.21.15.32:0/2999860454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.877+0000 7f5a0ad9d640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.877+0000 7f5a0b59e640 1 -- 172.21.15.32:0/2999860454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 msgr2=0x7f5a0c073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.877+0000 7f5a0b59e640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.877+0000 7f5a0b59e640 1 -- 172.21.15.32:0/2999860454 >> v1:172.21.15.32:6789/0 conn(0x7f5a0c0725e0 legacy=0x7f5a0c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.877+0000 7f5a0b59e640 1 -- 172.21.15.32:0/2999860454 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c15da50 con 0x7f5a0c15a420 2024-03-16T20:07:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.885+0000 7f5a0b59e640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c15a420 0x7f5a0c15a850 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009e50 tx=0x7f5a0401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe344a65bd67dab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a09d9b640 1 -- 172.21.15.32:0/2999860454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040a5830 con 0x7f5a0c15a420 2024-03-16T20:07:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a09d9b640 1 -- 172.21.15.32:0/2999860454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a040a5990 con 0x7f5a0c15a420 2024-03-16T20:07:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a09d9b640 1 -- 172.21.15.32:0/2999860454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040a5d20 con 0x7f5a0c15a420 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 -- 172.21.15.32:0/2999860454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c15a420 msgr2=0x7f5a0c15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c15a420 0x7f5a0c15a850 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009e50 tx=0x7f5a0401c970 comp rx=0 tx=0).stop 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 -- 172.21.15.32:0/2999860454 shutdown_connections 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c15a420 0x7f5a0c15a850 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 --2- 172.21.15.32:0/2999860454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 -- 172.21.15.32:0/2999860454 >> 172.21.15.32:0/2999860454 conn(0x7f5a0c06c0d0 msgr2=0x7f5a0c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 -- 172.21.15.32:0/2999860454 shutdown_connections 2024-03-16T20:07:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.889+0000 7f5a11093640 1 -- 172.21.15.32:0/2999860454 wait complete. 2024-03-16T20:07:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 Processor -- start 2024-03-16T20:07:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 -- start start 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c1f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0c15a420 0x7f5a0c1f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c1f2550 con 0x7f5a0c0725e0 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c1f2830 con 0x7f5a0c15a420 2024-03-16T20:07:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a11093640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c1f2b10 con 0x7f5a0c150d70 2024-03-16T20:07:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a0ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a0b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0c15a420 0x7f5a0c1f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a0ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41808/0 (socket says 172.21.15.32:41808) 2024-03-16T20:07:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.893+0000 7f5a0ad9d640 1 -- 172.21.15.32:0/517230932 learned_addr learned my addr 172.21.15.32:0/517230932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.897+0000 7f5a0a59c640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c1f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.901+0000 7f5a0ad9d640 1 -- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 msgr2=0x7f5a0c1f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.901+0000 7f5a0ad9d640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c1f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.901+0000 7f5a0ad9d640 1 -- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0c15a420 msgr2=0x7f5a0c1f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.901+0000 7f5a0ad9d640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0c15a420 0x7f5a0c1f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.901+0000 7f5a0ad9d640 1 -- 172.21.15.32:0/517230932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04009b30 con 0x7f5a0c0725e0 2024-03-16T20:07:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.905+0000 7f5a0ad9d640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c7f0 tx=0x7f59fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc004220 con 0x7f5a0c0725e0 2024-03-16T20:07:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc0043a0 con 0x7f5a0c0725e0 2024-03-16T20:07:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc004510 con 0x7f5a0c0725e0 2024-03-16T20:07:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f5a11093640 1 -- 172.21.15.32:0/517230932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c15d560 con 0x7f5a0c0725e0 2024-03-16T20:07:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f5a11093640 1 -- 172.21.15.32:0/517230932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c1f30d0 con 0x7f5a0c0725e0 2024-03-16T20:07:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:24.909+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e40052a0 con 0x7f5a0c0725e0 2024-03-16T20:07:25.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.197+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f59fc003a70 con 0x7f5a0c0725e0 2024-03-16T20:07:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.197+0000 7f59f37fe640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 0x7f59f8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.197+0000 7f5a0a59c640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 0x7f59f8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.201+0000 7f5a0a59c640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 0x7f59f8069350 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c15b190 tx=0x7f5a00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:25.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.233+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f59fc0559b0 con 0x7f5a0c0725e0 2024-03-16T20:07:25.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.233+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f59fc051c50 con 0x7f5a0c0725e0 2024-03-16T20:07:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.521+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e4005900 con 0x7f5a0c0725e0 2024-03-16T20:07:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.545+0000 7f59f37fe640 1 -- 172.21.15.32:0/517230932 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f59e4005900 con 0x7f5a0c0725e0 2024-03-16T20:07:25.535 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:25.535 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 msgr2=0x7f59f8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 0x7f59f8069350 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c15b190 tx=0x7f5a00009040 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 msgr2=0x7f5a0c1471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c7f0 tx=0x7f59fc00ccb0 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 shutdown_connections 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f59f8066e90 0x7f59f8069350 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a0c15a420 0x7f5a0c1f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5a0c150d70 0x7f5a0c1f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 --2- 172.21.15.32:0/517230932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a0c0725e0 0x7f5a0c1471c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 >> 172.21.15.32:0/517230932 conn(0x7f5a0c06c0d0 msgr2=0x7f5a0c14a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 shutdown_connections 2024-03-16T20:07:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:25.549+0000 7f59f17fa640 1 -- 172.21.15.32:0/517230932 wait complete. 2024-03-16T20:07:26.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:26.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:27.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 Processor -- start 2024-03-16T20:07:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 -- start start 2024-03-16T20:07:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc58150d70 0x7fdc58073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc580725e0 0x7fdc58070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc58072dd0 con 0x7fdc580725e0 2024-03-16T20:07:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5815d790 con 0x7fdc580708a0 2024-03-16T20:07:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc60455640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5815da50 con 0x7fdc58150d70 2024-03-16T20:07:27.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc5e9f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc5d9f0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc580725e0 0x7fdc58070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc5e1f1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc58150d70 0x7fdc58073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc5e9f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:59596/0 (socket says 172.21.15.32:59596) 2024-03-16T20:07:27.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.277+0000 7fdc5e9f2640 1 -- 172.21.15.32:0/1926160400 learned_addr learned my addr 172.21.15.32:0/1926160400 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5d9f0640 1 -- 172.21.15.32:0/1926160400 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc58150d70 msgr2=0x7fdc58073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5d9f0640 1 --2- 172.21.15.32:0/1926160400 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc58150d70 0x7fdc58073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5d9f0640 1 -- 172.21.15.32:0/1926160400 >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 msgr2=0x7fdc5815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5d9f0640 1 --2- 172.21.15.32:0/1926160400 >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5d9f0640 1 -- 172.21.15.32:0/1926160400 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5815dd30 con 0x7fdc580725e0 2024-03-16T20:07:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.281+0000 7fdc5e9f2640 1 --2- 172.21.15.32:0/1926160400 >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:07:27.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.285+0000 7fdc5d9f0640 1 --2- 172.21.15.32:0/1926160400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc580725e0 0x7fdc58070360 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdc5806e2d0 tx=0x7fdc4801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f84f635466d4002 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.289+0000 7fdc5d1ef640 1 -- 172.21.15.32:0/1926160400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc480ac410 con 0x7fdc580725e0 2024-03-16T20:07:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.289+0000 7fdc5d1ef640 1 -- 172.21.15.32:0/1926160400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc480ac570 con 0x7fdc580725e0 2024-03-16T20:07:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.289+0000 7fdc5d1ef640 1 -- 172.21.15.32:0/1926160400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc480ac900 con 0x7fdc580725e0 2024-03-16T20:07:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- 172.21.15.32:0/1926160400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc580725e0 msgr2=0x7fdc58070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- 172.21.15.32:0/1926160400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc580725e0 0x7fdc58070360 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdc5806e2d0 tx=0x7fdc4801c3f0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- 172.21.15.32:0/1926160400 shutdown_connections 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- 172.21.15.32:0/1926160400 >> v2:172.21.15.59:3300/0 conn(0x7fdc580708a0 0x7fdc5815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- 172.21.15.32:0/1926160400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc580725e0 0x7fdc58070360 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- 172.21.15.32:0/1926160400 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc58150d70 0x7fdc58073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- 172.21.15.32:0/1926160400 >> 172.21.15.32:0/1926160400 conn(0x7fdc5806c0d0 msgr2=0x7fdc5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- 172.21.15.32:0/1926160400 shutdown_connections 2024-03-16T20:07:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- 172.21.15.32:0/1926160400 wait complete. 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 Processor -- start 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- start start 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc580708a0 0x7fdc5815b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 0x7fdc58142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 0x7fdc58143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581f3ba0 con 0x7fdc58150d70 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5815c1f0 con 0x7fdc580725e0 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc60455640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5815c4d0 con 0x7fdc580708a0 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc5d9f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 0x7fdc58142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.293+0000 7fdc5e9f2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 0x7fdc58143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.297+0000 7fdc5e1f1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc580708a0 0x7fdc5815b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.297+0000 7fdc5d9f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 0x7fdc58142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:59604/0 (socket says 172.21.15.32:59604) 2024-03-16T20:07:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.297+0000 7fdc5d9f0640 1 -- 172.21.15.32:0/2807264172 learned_addr learned my addr 172.21.15.32:0/2807264172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.301+0000 7fdc5e9f2640 1 -- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc580708a0 msgr2=0x7fdc5815b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.301+0000 7fdc5e9f2640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc580708a0 0x7fdc5815b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.301+0000 7fdc5e9f2640 1 -- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 msgr2=0x7fdc58142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.301+0000 7fdc5e9f2640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 0x7fdc58142ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.301+0000 7fdc5e9f2640 1 -- 172.21.15.32:0/2807264172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50009b30 con 0x7fdc58150d70 2024-03-16T20:07:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.305+0000 7fdc5e9f2640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 0x7fdc58143420 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000eed0 tx=0x7fdc5000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50018070 con 0x7fdc58150d70 2024-03-16T20:07:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5000bb10 con 0x7fdc58150d70 2024-03-16T20:07:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48009420 con 0x7fdc58150d70 2024-03-16T20:07:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50010040 con 0x7fdc58150d70 2024-03-16T20:07:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5815d560 con 0x7fdc58150d70 2024-03-16T20:07:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.309+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc58068000 con 0x7fdc58150d70 2024-03-16T20:07:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.593+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdc500039a0 con 0x7fdc58150d70 2024-03-16T20:07:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.593+0000 7fdc4effd640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 0x7fdc3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.593+0000 7fdc5e1f1640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 0x7fdc3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.593+0000 7fdc5e1f1640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 0x7fdc3c069300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009ac0 tx=0x7fdc54009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.613+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdc50055c90 con 0x7fdc58150d70 2024-03-16T20:07:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.613+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdc50051f30 con 0x7fdc58150d70 2024-03-16T20:07:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.901+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc5806c4b0 con 0x7fdc58150d70 2024-03-16T20:07:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.917+0000 7fdc4effd640 1 -- 172.21.15.32:0/2807264172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fdc500555f0 con 0x7fdc58150d70 2024-03-16T20:07:27.907 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:27.907 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:27.907 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 msgr2=0x7fdc3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 0x7fdc3c069300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009ac0 tx=0x7fdc54009190 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 msgr2=0x7fdc58143420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 0x7fdc58143420 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000eed0 tx=0x7fdc5000b480 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 shutdown_connections 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdc3c066e40 0x7fdc3c069300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc58150d70 0x7fdc58143420 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc580725e0 0x7fdc58142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 --2- 172.21.15.32:0/2807264172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdc580708a0 0x7fdc5815b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 >> 172.21.15.32:0/2807264172 conn(0x7fdc5806c0d0 msgr2=0x7fdc580c4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 shutdown_connections 2024-03-16T20:07:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:27.921+0000 7fdc60455640 1 -- 172.21.15.32:0/2807264172 wait complete. 2024-03-16T20:07:28.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:28.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:29.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 Processor -- start 2024-03-16T20:07:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 -- start start 2024-03-16T20:07:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a601540c0 0x7f0a6015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0a600c3570 0x7f0a600c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6006c460 con 0x7f0a6015ac10 2024-03-16T20:07:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a600c3f40 con 0x7f0a601540c0 2024-03-16T20:07:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a667f9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6015a240 con 0x7f0a600c3570 2024-03-16T20:07:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a5ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a601540c0 0x7f0a6015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a5ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a601540c0 0x7f0a6015a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54796/0 (socket says 172.21.15.32:54796) 2024-03-16T20:07:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2739725717 learned_addr learned my addr 172.21.15.32:0/2739725717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.021+0000 7f0a64d96640 1 --2- 172.21.15.32:0/2739725717 >> v2:172.21.15.59:3300/0 conn(0x7f0a600c3570 0x7f0a600c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.025+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2739725717 >> v2:172.21.15.59:3300/0 conn(0x7f0a600c3570 msgr2=0x7f0a600c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.025+0000 7f0a5ffff640 1 --2- 172.21.15.32:0/2739725717 >> v2:172.21.15.59:3300/0 conn(0x7f0a600c3570 0x7f0a600c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.025+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2739725717 >> v1:172.21.15.59:6789/0 conn(0x7f0a6015ac10 legacy=0x7f0a600c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.025+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2739725717 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6006cf20 con 0x7f0a601540c0 2024-03-16T20:07:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.029+0000 7f0a5ffff640 1 --2- 172.21.15.32:0/2739725717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a601540c0 0x7f0a6015a6d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0a6006c740 tx=0x7f0a5001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b925e7ce6a96784c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a5effd640 1 -- 172.21.15.32:0/2739725717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500aa650 con 0x7f0a601540c0 2024-03-16T20:07:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a5effd640 1 -- 172.21.15.32:0/2739725717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500aa7b0 con 0x7f0a601540c0 2024-03-16T20:07:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a5effd640 1 -- 172.21.15.32:0/2739725717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500aab40 con 0x7f0a601540c0 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 -- 172.21.15.32:0/2739725717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a601540c0 msgr2=0x7f0a6015a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2739725717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a601540c0 0x7f0a6015a6d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0a6006c740 tx=0x7f0a5001c4f0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 -- 172.21.15.32:0/2739725717 shutdown_connections 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2739725717 >> v2:172.21.15.59:3300/0 conn(0x7f0a600c3570 0x7f0a600c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2739725717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a601540c0 0x7f0a6015a6d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 -- 172.21.15.32:0/2739725717 >> 172.21.15.32:0/2739725717 conn(0x7f0a60070f90 msgr2=0x7f0a600713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 -- 172.21.15.32:0/2739725717 shutdown_connections 2024-03-16T20:07:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.037+0000 7f0a667f9640 1 -- 172.21.15.32:0/2739725717 wait complete. 2024-03-16T20:07:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 Processor -- start 2024-03-16T20:07:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 -- start start 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a601540c0 0x7f0a600cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015ac10 0x7f0a600ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a601f68c0 con 0x7f0a600c3570 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6014ce40 con 0x7f0a6015ac10 2024-03-16T20:07:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a667f9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a601f5cb0 con 0x7f0a601540c0 2024-03-16T20:07:30.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a5ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a5ffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54802/0 (socket says 172.21.15.32:54802) 2024-03-16T20:07:30.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2212162135 learned_addr learned my addr 172.21.15.32:0/2212162135 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:30.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a64d96640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015ac10 0x7f0a600ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.041+0000 7f0a5f7fe640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a601540c0 0x7f0a600cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.045+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a601540c0 msgr2=0x7f0a600cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.045+0000 7f0a5ffff640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a601540c0 0x7f0a600cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.045+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015ac10 msgr2=0x7f0a600ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.045+0000 7f0a5ffff640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015ac10 0x7f0a600ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.045+0000 7f0a5ffff640 1 -- 172.21.15.32:0/2212162135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a500094d0 con 0x7f0a600c3570 2024-03-16T20:07:30.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.049+0000 7f0a5ffff640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0a50019100 tx=0x7f0a5001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:30.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500b3800 con 0x7f0a600c3570 2024-03-16T20:07:30.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500b3960 con 0x7f0a600c3570 2024-03-16T20:07:30.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a601f5f90 con 0x7f0a600c3570 2024-03-16T20:07:30.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500b3cf0 con 0x7f0a600c3570 2024-03-16T20:07:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a601f6550 con 0x7f0a600c3570 2024-03-16T20:07:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.057+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60070390 con 0x7f0a600c3570 2024-03-16T20:07:30.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.089+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0a500c7420 con 0x7f0a600c3570 2024-03-16T20:07:30.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.089+0000 7f0a5cff9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 0x7f0a4c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:30.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.089+0000 7f0a5f7fe640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 0x7f0a4c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:30.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.089+0000 7f0a5f7fe640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 0x7f0a4c069350 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0a60071a10 tx=0x7f0a54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.389+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0a500aa5d0 con 0x7f0a600c3570 2024-03-16T20:07:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.393+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0a500ec420 con 0x7f0a600c3570 2024-03-16T20:07:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.593+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a60068000 con 0x7f0a600c3570 2024-03-16T20:07:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.609+0000 7f0a5cff9640 1 -- 172.21.15.32:0/2212162135 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f0a500efae0 con 0x7f0a600c3570 2024-03-16T20:07:30.598 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:30.599 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 msgr2=0x7f0a4c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 0x7f0a4c069350 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0a60071a10 tx=0x7f0a54009040 comp rx=0 tx=0).stop 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 msgr2=0x7f0a60149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0a50019100 tx=0x7f0a5001ad30 comp rx=0 tx=0).stop 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 shutdown_connections 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0a4c066e90 0x7f0a4c069350 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a6015ac10 0x7f0a600ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0a601540c0 0x7f0a600cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 --2- 172.21.15.32:0/2212162135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a600c3570 0x7f0a60149ad0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 >> 172.21.15.32:0/2212162135 conn(0x7f0a60070f90 msgr2=0x7f0a6006efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 shutdown_connections 2024-03-16T20:07:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:30.613+0000 7f0a667f9640 1 -- 172.21.15.32:0/2212162135 wait complete. 2024-03-16T20:07:31.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:31.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:32.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.341+0000 7fb732344640 1 Processor -- start 2024-03-16T20:07:32.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.341+0000 7fb732344640 1 -- start start 2024-03-16T20:07:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb732344640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb72c0c36b0 0x7fb72c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb732344640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0c8ec0 con 0x7fb72c0c8980 2024-03-16T20:07:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb732344640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c072950 con 0x7fb72c151000 2024-03-16T20:07:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb732344640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c072f10 con 0x7fb72c0c36b0 2024-03-16T20:07:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb72b7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb72c0c36b0 0x7fb72c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:32.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb72bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb72c151000 0x7fb72c0c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:59606/0 (socket says 172.21.15.32:59606) 2024-03-16T20:07:32.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.345+0000 7fb72bfff640 1 -- 172.21.15.32:0/3507345329 learned_addr learned my addr 172.21.15.32:0/3507345329 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:32.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.349+0000 7fb72b7fe640 1 -- 172.21.15.32:0/3507345329 >> v1:172.21.15.59:6789/0 conn(0x7fb72c151000 legacy=0x7fb72c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.349+0000 7fb72b7fe640 1 -- 172.21.15.32:0/3507345329 >> v1:172.21.15.32:6789/0 conn(0x7fb72c0c8980 legacy=0x7fb72c1615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.349+0000 7fb72b7fe640 1 -- 172.21.15.32:0/3507345329 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c072c30 con 0x7fb72c0c36b0 2024-03-16T20:07:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.353+0000 7fb72b7fe640 1 --2- 172.21.15.32:0/3507345329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c36b0 0x7fb72c0c8420 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7200097c0 tx=0x7fb72001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d87e8d0356c6e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb72affd640 1 -- 172.21.15.32:0/3507345329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200ab600 con 0x7fb72c0c36b0 2024-03-16T20:07:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb72affd640 1 -- 172.21.15.32:0/3507345329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7200a7050 con 0x7fb72c0c36b0 2024-03-16T20:07:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb72affd640 1 -- 172.21.15.32:0/3507345329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200ab980 con 0x7fb72c0c36b0 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- 172.21.15.32:0/3507345329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c36b0 msgr2=0x7fb72c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 --2- 172.21.15.32:0/3507345329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c36b0 0x7fb72c0c8420 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7200097c0 tx=0x7fb72001c4a0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- 172.21.15.32:0/3507345329 shutdown_connections 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 --2- 172.21.15.32:0/3507345329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c36b0 0x7fb72c0c8420 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- 172.21.15.32:0/3507345329 >> 172.21.15.32:0/3507345329 conn(0x7fb72c06c0d0 msgr2=0x7fb72c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- 172.21.15.32:0/3507345329 shutdown_connections 2024-03-16T20:07:32.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- 172.21.15.32:0/3507345329 wait complete. 2024-03-16T20:07:32.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 Processor -- start 2024-03-16T20:07:32.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.361+0000 7fb732344640 1 -- start start 2024-03-16T20:07:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c0c36b0 0x7fb72c147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c151000 0x7fb72c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c1484a0 con 0x7fb72c0c8980 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c149330 con 0x7fb72c151000 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb732344640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c14a720 con 0x7fb72c0c36b0 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54814/0 (socket says 172.21.15.32:54814) 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb72b7fe640 1 -- 172.21.15.32:0/217913435 learned_addr learned my addr 172.21.15.32:0/217913435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:32.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.365+0000 7fb72bfff640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c0c36b0 0x7fb72c147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:32.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb7308e1640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c151000 0x7fb72c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:32.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb72b7fe640 1 -- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c0c36b0 msgr2=0x7fb72c147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb72b7fe640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c0c36b0 0x7fb72c147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb72b7fe640 1 -- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c151000 msgr2=0x7fb72c156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb72b7fe640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c151000 0x7fb72c156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.369+0000 7fb72b7fe640 1 -- 172.21.15.32:0/217913435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009420 con 0x7fb72c0c8980 2024-03-16T20:07:32.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.373+0000 7fb72b7fe640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb7200ae040 tx=0x7fb720019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:32.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200b4840 con 0x7fb72c0c8980 2024-03-16T20:07:32.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7200b49c0 con 0x7fb72c0c8980 2024-03-16T20:07:32.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200b4d70 con 0x7fb72c0c8980 2024-03-16T20:07:32.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb732344640 1 -- 172.21.15.32:0/217913435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c14a1a0 con 0x7fb72c0c8980 2024-03-16T20:07:32.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb732344640 1 -- 172.21.15.32:0/217913435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c155fe0 con 0x7fb72c0c8980 2024-03-16T20:07:32.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.377+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7040052a0 con 0x7fb72c0c8980 2024-03-16T20:07:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.405+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb7200b2040 con 0x7fb72c0c8980 2024-03-16T20:07:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.405+0000 7fb728ff9640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 0x7fb718069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.405+0000 7fb72bfff640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 0x7fb718069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.405+0000 7fb72bfff640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 0x7fb718069350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb71c005890 tx=0x7fb71c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:32.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.409+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb7200f0360 con 0x7fb72c0c8980 2024-03-16T20:07:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.673+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb7200ebf00 con 0x7fb72c0c8980 2024-03-16T20:07:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.897+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb704005900 con 0x7fb72c0c8980 2024-03-16T20:07:32.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:32.997+0000 7fb728ff9640 1 -- 172.21.15.32:0/217913435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fb704005900 con 0x7fb72c0c8980 2024-03-16T20:07:32.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:32.991 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:32.991 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 msgr2=0x7fb718069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 0x7fb718069350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb71c005890 tx=0x7fb71c009040 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 msgr2=0x7fb72c14d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb7200ae040 tx=0x7fb720019240 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 shutdown_connections 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb718066e90 0x7fb718069350 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c151000 0x7fb72c156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb72c0c8980 0x7fb72c14d6e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 --2- 172.21.15.32:0/217913435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c0c36b0 0x7fb72c147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:32.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 >> 172.21.15.32:0/217913435 conn(0x7fb72c06c0d0 msgr2=0x7fb72c1597d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:32.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 shutdown_connections 2024-03-16T20:07:32.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:33.001+0000 7fb7127fc640 1 -- 172.21.15.32:0/217913435 wait complete. 2024-03-16T20:07:33.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:33.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:34.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:34.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.737+0000 7fc7f01c9640 1 Processor -- start 2024-03-16T20:07:34.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.737+0000 7fc7f01c9640 1 -- start start 2024-03-16T20:07:34.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7f01c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7e815ab30 0x7fc7e8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7f01c9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e8072dd0 con 0x7fc7e8150d70 2024-03-16T20:07:34.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7f01c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e815d790 con 0x7fc7e80725e0 2024-03-16T20:07:34.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7f01c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e815da50 con 0x7fc7e815ab30 2024-03-16T20:07:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7edf65640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7e8150d70 0x7fc7e8070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56894/0 (socket says 172.21.15.32:56894) 2024-03-16T20:07:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7edf65640 1 -- 172.21.15.32:0/2426215780 learned_addr learned my addr 172.21.15.32:0/2426215780 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.741+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2426215780 >> v2:172.21.15.59:3300/0 conn(0x7fc7e815ab30 0x7fc7e8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.745+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699693922 0 0) 0x7fc7e8072dd0 con 0x7fc7e8150d70 2024-03-16T20:07:34.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.745+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d4003530 con 0x7fc7e8150d70 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268227395 0 0) 0x7fc7d4003530 con 0x7fc7e8150d70 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 >> v2:172.21.15.59:3300/0 conn(0x7fc7e815ab30 msgr2=0x7fc7e8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 --2- 172.21.15.32:0/2426215780 >> v2:172.21.15.59:3300/0 conn(0x7fc7e815ab30 0x7fc7e8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 >> v1:172.21.15.59:6789/0 conn(0x7fc7e80725e0 legacy=0x7fc7e815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e815dd30 con 0x7fc7e8150d70 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fc7e4001680 con 0x7fc7e8150d70 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 >> v1:172.21.15.32:6789/0 conn(0x7fc7e8150d70 legacy=0x7fc7e8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40039d0 0x7fc7d4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7d40047d0 0x7fc7d4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 0x7fc7d400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e815da50 con 0x7fc7d40052b0 2024-03-16T20:07:34.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d400c1b0 con 0x7fc7d40039d0 2024-03-16T20:07:34.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d400c370 con 0x7fc7d40047d0 2024-03-16T20:07:34.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.749+0000 7fc7ed764640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7d40047d0 0x7fc7d4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.753+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 0x7fc7d400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.753+0000 7fc7edf65640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40039d0 0x7fc7d4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 -- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7d40047d0 msgr2=0x7fc7d4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7d40047d0 0x7fc7d4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 -- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40039d0 msgr2=0x7fc7d4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40039d0 0x7fc7d4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 -- 172.21.15.32:0/2426215780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4003530 con 0x7fc7d40052b0 2024-03-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.757+0000 7fc7ee766640 1 -- 172.21.15.32:0/2426215780 reap_dead start 2024-03-16T20:07:34.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.761+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 0x7fc7d400b990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009fc0 tx=0x7fc7e001cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2fbcf9c983ff96 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.801+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00a5c80 con 0x7fc7d40052b0 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.801+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e000f040 con 0x7fc7d40052b0 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.801+0000 7fc7ecf63640 1 -- 172.21.15.32:0/2426215780 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00039a0 con 0x7fc7d40052b0 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 msgr2=0x7fc7d400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 0x7fc7d400b990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009fc0 tx=0x7fc7e001cc90 comp rx=0 tx=0).stop 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2426215780 shutdown_connections 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2426215780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d40052b0 0x7fc7d400b990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2426215780 >> 172.21.15.32:0/2426215780 conn(0x7fc7e806c0d0 msgr2=0x7fc7e806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:34.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2426215780 shutdown_connections 2024-03-16T20:07:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2426215780 wait complete. 2024-03-16T20:07:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 Processor -- start 2024-03-16T20:07:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 -- start start 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40052b0 0x7fc7e8139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e815afb0 0x7fc7e8134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.805+0000 7fc7f01c9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7f01c9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80c5660 con 0x7fc7e815b430 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7f01c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80c5940 con 0x7fc7d40052b0 2024-03-16T20:07:34.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7f01c9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80c6290 con 0x7fc7e815afb0 2024-03-16T20:07:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7edf65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40052b0 0x7fc7e8139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7ed764640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e815afb0 0x7fc7e8134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7ee766640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7ee766640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54834/0 (socket says 172.21.15.32:54834) 2024-03-16T20:07:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.809+0000 7fc7ee766640 1 -- 172.21.15.32:0/2466898127 learned_addr learned my addr 172.21.15.32:0/2466898127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.813+0000 7fc7ee766640 1 -- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e815afb0 msgr2=0x7fc7e8134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.813+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e815afb0 0x7fc7e8134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.813+0000 7fc7ee766640 1 -- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40052b0 msgr2=0x7fc7e8139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.813+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40052b0 0x7fc7e8139740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.813+0000 7fc7ee766640 1 -- 172.21.15.32:0/2466898127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e815d230 con 0x7fc7e815b430 2024-03-16T20:07:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.825+0000 7fc7ee766640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc7e001c700 tx=0x7fc7e0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:34.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0003c60 con 0x7fc7e815b430 2024-03-16T20:07:34.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e0003dc0 con 0x7fc7e815b430 2024-03-16T20:07:34.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e80ca540 con 0x7fc7e815b430 2024-03-16T20:07:34.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0004230 con 0x7fc7e815b430 2024-03-16T20:07:34.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e81388f0 con 0x7fc7e815b430 2024-03-16T20:07:34.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.829+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e8072ff0 con 0x7fc7e815b430 2024-03-16T20:07:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.849+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc7e0004390 con 0x7fc7e815b430 2024-03-16T20:07:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.849+0000 7fc7de7fc640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 0x7fc7cc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.849+0000 7fc7edf65640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 0x7fc7cc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:34.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.849+0000 7fc7edf65640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 0x7fc7cc069380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc7e81383a0 tx=0x7fc7e4007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:34.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:34.853+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc7e00f1340 con 0x7fc7e815b430 2024-03-16T20:07:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.125+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc7e00ed5e0 con 0x7fc7e815b430 2024-03-16T20:07:35.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.353+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7e80722b0 con 0x7fc7e815b430 2024-03-16T20:07:35.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7de7fc640 1 -- 172.21.15.32:0/2466898127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc7e00f0ca0 con 0x7fc7e815b430 2024-03-16T20:07:35.442 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:35.442 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:35.442 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 msgr2=0x7fc7cc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 0x7fc7cc069380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc7e81383a0 tx=0x7fc7e4007040 comp rx=0 tx=0).stop 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 msgr2=0x7fc7e8134f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc7e001c700 tx=0x7fc7e0009750 comp rx=0 tx=0).stop 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 shutdown_connections 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc7cc066ec0 0x7fc7cc069380 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7e815b430 0x7fc7e8134f80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc7e815afb0 0x7fc7e8134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 --2- 172.21.15.32:0/2466898127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7d40052b0 0x7fc7e8139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 >> 172.21.15.32:0/2466898127 conn(0x7fc7e806c0d0 msgr2=0x7fc7e80c84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 shutdown_connections 2024-03-16T20:07:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:35.453+0000 7fc7f01c9640 1 -- 172.21.15.32:0/2466898127 wait complete. 2024-03-16T20:07:36.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:36.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:37.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:37.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 Processor -- start 2024-03-16T20:07:37.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 -- start start 2024-03-16T20:07:37.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4e015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e015ad50 con 0x7fa4e0153fc0 2024-03-16T20:07:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e015a1f0 con 0x7fa4e015df30 2024-03-16T20:07:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e77ca640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00c7ce0 con 0x7fa4e00c8b90 2024-03-16T20:07:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e5566640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa4e0153fc0 0x7fa4e00c72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56910/0 (socket says 172.21.15.32:56910) 2024-03-16T20:07:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e5566640 1 -- 172.21.15.32:0/814268910 learned_addr learned my addr 172.21.15.32:0/814268910 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.197+0000 7fa4e4d65640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4e015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.201+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075222030 0 0) 0x7fa4e015ad50 con 0x7fa4e0153fc0 2024-03-16T20:07:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.201+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc003530 con 0x7fa4e0153fc0 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809468136 0 0) 0x7fa4cc003530 con 0x7fa4e0153fc0 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 msgr2=0x7fa4e015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4e015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 >> v1:172.21.15.59:6789/0 conn(0x7fa4e015df30 legacy=0x7fa4e00c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e015a4b0 con 0x7fa4e0153fc0 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fa4dc001680 con 0x7fa4e0153fc0 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 >> v1:172.21.15.32:6789/0 conn(0x7fa4e0153fc0 legacy=0x7fa4e00c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0039b0 0x7fa4cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4cc0047d0 0x7fa4cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00bb00 con 0x7fa4cc0039b0 2024-03-16T20:07:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00bc70 con 0x7fa4cc0047d0 2024-03-16T20:07:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.205+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00be30 con 0x7fa4e00c8b90 2024-03-16T20:07:37.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.209+0000 7fa4e5566640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0039b0 0x7fa4cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.209+0000 7fa4e4d65640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4cc0047d0 0x7fa4cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5d67640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5566640 1 -- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 msgr2=0x7fa4cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5566640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4e00c8b90 0x7fa4cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5566640 1 -- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4cc0047d0 msgr2=0x7fa4cc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:37.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5566640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4cc0047d0 0x7fa4cc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.213+0000 7fa4e5566640 1 -- 172.21.15.32:0/814268910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc003530 con 0x7fa4cc0039b0 2024-03-16T20:07:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.217+0000 7fa4e5566640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0039b0 0x7fa4cc003dd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc007fc0 tx=0x7fa4dc01f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35c0eb4342606f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0058e0 con 0x7fa4cc0039b0 2024-03-16T20:07:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc005a60 con 0x7fa4cc0039b0 2024-03-16T20:07:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4d7fff640 1 -- 172.21.15.32:0/814268910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc005e10 con 0x7fa4cc0039b0 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 -- 172.21.15.32:0/814268910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0039b0 msgr2=0x7fa4cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/814268910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0039b0 0x7fa4cc003dd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc007fc0 tx=0x7fa4dc01f2b0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e5d67640 1 -- 172.21.15.32:0/814268910 reap_dead start 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 -- 172.21.15.32:0/814268910 shutdown_connections 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 -- 172.21.15.32:0/814268910 >> 172.21.15.32:0/814268910 conn(0x7fa4e00cc7a0 msgr2=0x7fa4e00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 -- 172.21.15.32:0/814268910 shutdown_connections 2024-03-16T20:07:37.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.221+0000 7fa4e77ca640 1 -- 172.21.15.32:0/814268910 wait complete. 2024-03-16T20:07:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 Processor -- start 2024-03-16T20:07:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 -- start start 2024-03-16T20:07:37.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4cc0039b0 0x7fa4e01efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e00c6d50 0x7fa4e00cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0154b40 con 0x7fa4e01f3060 2024-03-16T20:07:37.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0155f20 con 0x7fa4e00c6d50 2024-03-16T20:07:37.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e77ca640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01f0180 con 0x7fa4cc0039b0 2024-03-16T20:07:37.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e5d67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e5d67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54848/0 (socket says 172.21.15.32:54848) 2024-03-16T20:07:37.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e5d67640 1 -- 172.21.15.32:0/3350754351 learned_addr learned my addr 172.21.15.32:0/3350754351 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:37.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e4d65640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e00c6d50 0x7fa4e00cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.225+0000 7fa4e5566640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4cc0039b0 0x7fa4e01efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.229+0000 7fa4e5d67640 1 -- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4cc0039b0 msgr2=0x7fa4e01efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.229+0000 7fa4e5d67640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4cc0039b0 0x7fa4e01efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.229+0000 7fa4e5d67640 1 -- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e00c6d50 msgr2=0x7fa4e00cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.229+0000 7fa4e5d67640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e00c6d50 0x7fa4e00cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.229+0000 7fa4e5d67640 1 -- 172.21.15.32:0/3350754351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e01f0a00 con 0x7fa4e01f3060 2024-03-16T20:07:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.233+0000 7fa4e5d67640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8000f90 tx=0x7fa4d800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800ee90 con 0x7fa4e01f3060 2024-03-16T20:07:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d8006b60 con 0x7fa4e01f3060 2024-03-16T20:07:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01f0440 con 0x7fa4e01f3060 2024-03-16T20:07:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8003980 con 0x7fa4e01f3060 2024-03-16T20:07:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01547d0 con 0x7fa4e01f3060 2024-03-16T20:07:37.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.237+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0154cb0 con 0x7fa4e01f3060 2024-03-16T20:07:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.281+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa4d8003ae0 con 0x7fa4e01f3060 2024-03-16T20:07:37.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.281+0000 7fa4d5ffb640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 0x7fa4c4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:37.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.281+0000 7fa4e5566640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 0x7fa4c4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:37.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.281+0000 7fa4e5566640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 0x7fa4c4069330 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0ae000 tx=0x7fa4dc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.561+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa4d80551d0 con 0x7fa4e01f3060 2024-03-16T20:07:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.561+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa4d8051470 con 0x7fa4e01f3060 2024-03-16T20:07:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.813+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4e0068000 con 0x7fa4e01f3060 2024-03-16T20:07:37.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4d5ffb640 1 -- 172.21.15.32:0/3350754351 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa4d8019020 con 0x7fa4e01f3060 2024-03-16T20:07:37.874 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:37.874 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:37.875 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 msgr2=0x7fa4c4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 0x7fa4c4069330 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0ae000 tx=0x7fa4dc0aa070 comp rx=0 tx=0).stop 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 msgr2=0x7fa4e0154fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8000f90 tx=0x7fa4d800eb50 comp rx=0 tx=0).stop 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 shutdown_connections 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa4c4066e70 0x7fa4c4069330 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4e01f3060 0x7fa4e0154fd0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e00c6d50 0x7fa4e00cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 --2- 172.21.15.32:0/3350754351 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa4cc0039b0 0x7fa4e01efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 >> 172.21.15.32:0/3350754351 conn(0x7fa4e00cc7a0 msgr2=0x7fa4e006b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 shutdown_connections 2024-03-16T20:07:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:37.889+0000 7fa4e77ca640 1 -- 172.21.15.32:0/3350754351 wait complete. 2024-03-16T20:07:38.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:38.504 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:39.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.625+0000 7f88f4ef0640 1 Processor -- start 2024-03-16T20:07:39.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.625+0000 7f88f4ef0640 1 -- start start 2024-03-16T20:07:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88f4ef0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f0153fa0 0x7f88f0070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88f4ef0640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88f00c7470 0x7f88f015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88f4ef0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f00c8590 con 0x7f88f015d920 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88f4ef0640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00c3f00 con 0x7f88f00c7470 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88f4ef0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0070650 con 0x7f88f0153fa0 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88edd9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88f00c7470 0x7f88f015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88eed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f88f015d920 0x7f88f00724a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:51688/0 (socket says 172.21.15.32:51688) 2024-03-16T20:07:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88eed9d640 1 -- 172.21.15.32:0/3872777287 learned_addr learned my addr 172.21.15.32:0/3872777287 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.629+0000 7f88ee59c640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f0153fa0 0x7f88f0070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.633+0000 7f88edd9b640 1 -- 172.21.15.32:0/3872777287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f0153fa0 msgr2=0x7f88f0070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.633+0000 7f88edd9b640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f0153fa0 0x7f88f0070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.633+0000 7f88edd9b640 1 -- 172.21.15.32:0/3872777287 >> v1:172.21.15.59:6789/0 conn(0x7f88f015d920 legacy=0x7f88f00724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.633+0000 7f88edd9b640 1 -- 172.21.15.32:0/3872777287 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0073590 con 0x7f88f00c7470 2024-03-16T20:07:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.633+0000 7f88edd9b640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f00c7470 0x7f88f015d3e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f88e40097c0 tx=0x7f88e401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33343d72088bd0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:39.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88ed59a640 1 -- 172.21.15.32:0/3872777287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a8990 con 0x7f88f00c7470 2024-03-16T20:07:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88ed59a640 1 -- 172.21.15.32:0/3872777287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e40a8af0 con 0x7f88f00c7470 2024-03-16T20:07:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88ed59a640 1 -- 172.21.15.32:0/3872777287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a8e80 con 0x7f88f00c7470 2024-03-16T20:07:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 -- 172.21.15.32:0/3872777287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f00c7470 msgr2=0x7f88f015d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:39.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f00c7470 0x7f88f015d3e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f88e40097c0 tx=0x7f88e401ec90 comp rx=0 tx=0).stop 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 -- 172.21.15.32:0/3872777287 shutdown_connections 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f00c7470 0x7f88f015d3e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/3872777287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f0153fa0 0x7f88f0070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 -- 172.21.15.32:0/3872777287 >> 172.21.15.32:0/3872777287 conn(0x7f88f006c0d0 msgr2=0x7f88f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 -- 172.21.15.32:0/3872777287 shutdown_connections 2024-03-16T20:07:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.641+0000 7f88f4ef0640 1 -- 172.21.15.32:0/3872777287 wait complete. 2024-03-16T20:07:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 Processor -- start 2024-03-16T20:07:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 -- start start 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f00c7470 0x7f88f0157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f015d920 0x7f88f0154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0156100 con 0x7f88f0153fa0 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01ef470 con 0x7f88f015d920 2024-03-16T20:07:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88f4ef0640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01f0400 con 0x7f88f00c7470 2024-03-16T20:07:39.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88edd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88edd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41062/0 (socket says 172.21.15.32:41062) 2024-03-16T20:07:39.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88edd9b640 1 -- 172.21.15.32:0/1430176438 learned_addr learned my addr 172.21.15.32:0/1430176438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:39.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88ee59c640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f00c7470 0x7f88f0157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.645+0000 7f88eed9d640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f015d920 0x7f88f0154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.649+0000 7f88edd9b640 1 -- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f00c7470 msgr2=0x7f88f0157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:39.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.649+0000 7f88edd9b640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f00c7470 0x7f88f0157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.649+0000 7f88edd9b640 1 -- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f015d920 msgr2=0x7f88f0154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.649+0000 7f88edd9b640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f015d920 0x7f88f0154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.649+0000 7f88edd9b640 1 -- 172.21.15.32:0/1430176438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009420 con 0x7f88f0153fa0 2024-03-16T20:07:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.653+0000 7f88edd9b640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f88e40b3040 tx=0x7f88e401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.657+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b0e10 con 0x7f88f0153fa0 2024-03-16T20:07:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.657+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e40b9410 con 0x7f88f0153fa0 2024-03-16T20:07:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.657+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f015aa80 con 0x7f88f0153fa0 2024-03-16T20:07:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.657+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b97c0 con 0x7f88f0153fa0 2024-03-16T20:07:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.657+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f01ef920 con 0x7f88f0153fa0 2024-03-16T20:07:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.661+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f0068000 con 0x7f88f0153fa0 2024-03-16T20:07:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.681+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f88e40b7040 con 0x7f88f0153fa0 2024-03-16T20:07:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.681+0000 7f88deffd640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 0x7f88d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.681+0000 7f88ee59c640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 0x7f88d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.681+0000 7f88ee59c640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 0x7f88d0069300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009b60 tx=0x7f88e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:39.953+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f88e40f3e70 con 0x7f88f0153fa0 2024-03-16T20:07:40.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.053+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f88e40f0150 con 0x7f88f0153fa0 2024-03-16T20:07:40.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.253+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88f014a8f0 con 0x7f88f0153fa0 2024-03-16T20:07:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88deffd640 1 -- 172.21.15.32:0/1430176438 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f88e40b5070 con 0x7f88f0153fa0 2024-03-16T20:07:40.262 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:40.263 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 msgr2=0x7f88d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 0x7f88d0069300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009b60 tx=0x7f88e0009190 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 msgr2=0x7f88f0155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f88e40b3040 tx=0x7f88e401ba30 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 shutdown_connections 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88d0066e40 0x7f88d0069300 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f015d920 0x7f88f0154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88f0153fa0 0x7f88f0155bc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 --2- 172.21.15.32:0/1430176438 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88f00c7470 0x7f88f0157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 >> 172.21.15.32:0/1430176438 conn(0x7f88f006c0d0 msgr2=0x7f88f00c8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 shutdown_connections 2024-03-16T20:07:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:40.277+0000 7f88f4ef0640 1 -- 172.21.15.32:0/1430176438 wait complete. 2024-03-16T20:07:40.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:40.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:41.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.009+0000 7fc905ea4640 1 Processor -- start 2024-03-16T20:07:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.009+0000 7fc905ea4640 1 -- start start 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc905ea4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9000714f0 0x7fc9000ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc905ea4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc9000c2910 0x7fc9000cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc905ea4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000ca1d0 con 0x7fc900071110 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc905ea4640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000c34a0 con 0x7fc9000c2910 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc905ea4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000c9ba0 con 0x7fc9000714f0 2024-03-16T20:07:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc8fffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc9000c2910 0x7fc9000cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc8feffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9000714f0 0x7fc9000ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc8feffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9000714f0 0x7fc9000ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:40942/0 (socket says 172.21.15.32:40942) 2024-03-16T20:07:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.013+0000 7fc8feffd640 1 -- 172.21.15.32:0/3595814540 learned_addr learned my addr 172.21.15.32:0/3595814540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.017+0000 7fc8feffd640 1 -- 172.21.15.32:0/3595814540 >> v2:172.21.15.32:3300/0 conn(0x7fc9000c2910 msgr2=0x7fc9000cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.017+0000 7fc8feffd640 1 --2- 172.21.15.32:0/3595814540 >> v2:172.21.15.32:3300/0 conn(0x7fc9000c2910 0x7fc9000cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.017+0000 7fc8feffd640 1 -- 172.21.15.32:0/3595814540 >> v1:172.21.15.32:6789/0 conn(0x7fc900071110 legacy=0x7fc9000c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.017+0000 7fc8feffd640 1 -- 172.21.15.32:0/3595814540 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000c2ee0 con 0x7fc9000714f0 2024-03-16T20:07:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.021+0000 7fc8feffd640 1 --2- 172.21.15.32:0/3595814540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000ca740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc9000ce990 tx=0x7fc8f401ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b883a97ab5f2305 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.025+0000 7fc8fe7fc640 1 -- 172.21.15.32:0/3595814540 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40a8a10 con 0x7fc9000714f0 2024-03-16T20:07:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.025+0000 7fc8fe7fc640 1 -- 172.21.15.32:0/3595814540 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f40a8b70 con 0x7fc9000714f0 2024-03-16T20:07:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.025+0000 7fc8fe7fc640 1 -- 172.21.15.32:0/3595814540 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40b0440 con 0x7fc9000714f0 2024-03-16T20:07:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- 172.21.15.32:0/3595814540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 msgr2=0x7fc9000ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3595814540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000ca740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc9000ce990 tx=0x7fc8f401ed10 comp rx=0 tx=0).stop 2024-03-16T20:07:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- 172.21.15.32:0/3595814540 shutdown_connections 2024-03-16T20:07:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3595814540 >> v2:172.21.15.32:3300/0 conn(0x7fc9000c2910 0x7fc9000cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3595814540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000ca740 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- 172.21.15.32:0/3595814540 >> 172.21.15.32:0/3595814540 conn(0x7fc90006afc0 msgr2=0x7fc90006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- 172.21.15.32:0/3595814540 shutdown_connections 2024-03-16T20:07:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- 172.21.15.32:0/3595814540 wait complete. 2024-03-16T20:07:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 Processor -- start 2024-03-16T20:07:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- start start 2024-03-16T20:07:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9000c2910 0x7fc9001f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001f1b70 con 0x7fc900071110 2024-03-16T20:07:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90014d4c0 con 0x7fc9000714f0 2024-03-16T20:07:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.029+0000 7fc905ea4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90014d7a0 con 0x7fc9000c2910 2024-03-16T20:07:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.033+0000 7fc8ff7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.033+0000 7fc8feffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.033+0000 7fc8ff7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41074/0 (socket says 172.21.15.32:41074) 2024-03-16T20:07:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.033+0000 7fc8ff7fe640 1 -- 172.21.15.32:0/3781821653 learned_addr learned my addr 172.21.15.32:0/3781821653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.033+0000 7fc8fffff640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9000c2910 0x7fc9001f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.037+0000 7fc8ff7fe640 1 -- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9000c2910 msgr2=0x7fc9001f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.037+0000 7fc8ff7fe640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9000c2910 0x7fc9001f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.037+0000 7fc8ff7fe640 1 -- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 msgr2=0x7fc9000c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.037+0000 7fc8ff7fe640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.037+0000 7fc8ff7fe640 1 -- 172.21.15.32:0/3781821653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40094d0 con 0x7fc900071110 2024-03-16T20:07:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.041+0000 7fc8ff7fe640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000c420 tx=0x7fc8f000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.045+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f00052d0 con 0x7fc900071110 2024-03-16T20:07:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.045+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0005430 con 0x7fc900071110 2024-03-16T20:07:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.045+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0007260 con 0x7fc900071110 2024-03-16T20:07:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.049+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000b2e20 con 0x7fc900071110 2024-03-16T20:07:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.049+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9001edaf0 con 0x7fc900071110 2024-03-16T20:07:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.049+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9000b2c20 con 0x7fc900071110 2024-03-16T20:07:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.081+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc8f00073c0 con 0x7fc900071110 2024-03-16T20:07:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.081+0000 7fc8e7fff640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 0x7fc8ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.081+0000 7fc8feffd640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 0x7fc8ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:42.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.085+0000 7fc8feffd640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 0x7fc8ec069350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40b2000 tx=0x7fc8f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.389+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc8f0007670 con 0x7fc900071110 2024-03-16T20:07:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.389+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc8f0051530 con 0x7fc900071110 2024-03-16T20:07:42.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.609+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc900068000 con 0x7fc900071110 2024-03-16T20:07:42.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.621+0000 7fc8e7fff640 1 -- 172.21.15.32:0/3781821653 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fc8f0020020 con 0x7fc900071110 2024-03-16T20:07:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:42.608 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:42.608 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 msgr2=0x7fc8ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 0x7fc8ec069350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40b2000 tx=0x7fc8f4009040 comp rx=0 tx=0).stop 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 msgr2=0x7fc9001ee2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000c420 tx=0x7fc8f000c8e0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 shutdown_connections 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc8ec066e90 0x7fc8ec069350 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9000c2910 0x7fc9001f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9000714f0 0x7fc9000c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 --2- 172.21.15.32:0/3781821653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc900071110 0x7fc9001ee2f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:42.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 >> 172.21.15.32:0/3781821653 conn(0x7fc90006afc0 msgr2=0x7fc9000ca6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:42.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 shutdown_connections 2024-03-16T20:07:42.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:42.625+0000 7fc905ea4640 1 -- 172.21.15.32:0/3781821653 wait complete. 2024-03-16T20:07:42.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:07:42.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:07:43.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:43.233 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:44.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:44.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 Processor -- start 2024-03-16T20:07:44.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 -- start start 2024-03-16T20:07:44.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f815a420 0x7fa8f815a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f8072dd0 con 0x7fa8f8150d70 2024-03-16T20:07:44.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f815d790 con 0x7fa8f815a420 2024-03-16T20:07:44.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8fde2f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815ad90 con 0x7fa8f80725e0 2024-03-16T20:07:44.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8f77fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8f8150d70 0x7fa8f8070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46634/0 (socket says 172.21.15.32:46634) 2024-03-16T20:07:44.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8f77fe640 1 -- 172.21.15.32:0/3956412543 learned_addr learned my addr 172.21.15.32:0/3956412543 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:44.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8f6ffd640 1 --2- 172.21.15.32:0/3956412543 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.357+0000 7fa8f7fff640 1 --2- 172.21.15.32:0/3956412543 >> v2:172.21.15.59:3300/0 conn(0x7fa8f815a420 0x7fa8f815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.361+0000 7fa8f67fc640 1 -- 172.21.15.32:0/3956412543 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267260357 0 0) 0x7fa8f8072dd0 con 0x7fa8f8150d70 2024-03-16T20:07:44.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.361+0000 7fa8f67fc640 1 -- 172.21.15.32:0/3956412543 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0003530 con 0x7fa8f8150d70 2024-03-16T20:07:44.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.365+0000 7fa8f6ffd640 1 -- 172.21.15.32:0/3956412543 >> v2:172.21.15.59:3300/0 conn(0x7fa8f815a420 msgr2=0x7fa8f815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:44.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.365+0000 7fa8f6ffd640 1 --2- 172.21.15.32:0/3956412543 >> v2:172.21.15.59:3300/0 conn(0x7fa8f815a420 0x7fa8f815a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.365+0000 7fa8f6ffd640 1 -- 172.21.15.32:0/3956412543 >> v1:172.21.15.32:6789/0 conn(0x7fa8f8150d70 legacy=0x7fa8f8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:44.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.365+0000 7fa8f6ffd640 1 -- 172.21.15.32:0/3956412543 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f815da50 con 0x7fa8f80725e0 2024-03-16T20:07:44.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.373+0000 7fa8f6ffd640 1 --2- 172.21.15.32:0/3956412543 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f8073070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0097c0 tx=0x7fa8ec01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c8b9178ebd2f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8f67fc640 1 -- 172.21.15.32:0/3956412543 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0ac640 con 0x7fa8f80725e0 2024-03-16T20:07:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8f67fc640 1 -- 172.21.15.32:0/3956412543 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ec0ac7a0 con 0x7fa8f80725e0 2024-03-16T20:07:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8f67fc640 1 -- 172.21.15.32:0/3956412543 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0acb30 con 0x7fa8f80725e0 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 -- 172.21.15.32:0/3956412543 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 msgr2=0x7fa8f8073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/3956412543 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f8073070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0097c0 tx=0x7fa8ec01c3e0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 -- 172.21.15.32:0/3956412543 shutdown_connections 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/3956412543 >> v2:172.21.15.59:3300/0 conn(0x7fa8f815a420 0x7fa8f815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/3956412543 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f8073070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 -- 172.21.15.32:0/3956412543 >> 172.21.15.32:0/3956412543 conn(0x7fa8f806c0d0 msgr2=0x7fa8f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 -- 172.21.15.32:0/3956412543 shutdown_connections 2024-03-16T20:07:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.381+0000 7fa8fde2f640 1 -- 172.21.15.32:0/3956412543 wait complete. 2024-03-16T20:07:44.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 Processor -- start 2024-03-16T20:07:44.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 -- start start 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f81f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f8150d70 0x7fa8f814b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f8154250 con 0x7fa8f815a420 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815b130 con 0x7fa8f8150d70 2024-03-16T20:07:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8fde2f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815b6d0 con 0x7fa8f80725e0 2024-03-16T20:07:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41078/0 (socket says 172.21.15.32:41078) 2024-03-16T20:07:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8f7fff640 1 -- 172.21.15.32:0/1607394 learned_addr learned my addr 172.21.15.32:0/1607394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.385+0000 7fa8f6ffd640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f8150d70 0x7fa8f814b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f77fe640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f81f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f7fff640 1 -- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 msgr2=0x7fa8f81f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:44.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f7fff640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f81f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f7fff640 1 -- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f8150d70 msgr2=0x7fa8f814b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:44.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f7fff640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f8150d70 0x7fa8f814b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:44.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.389+0000 7fa8f7fff640 1 -- 172.21.15.32:0/1607394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec009420 con 0x7fa8f815a420 2024-03-16T20:07:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.393+0000 7fa8f7fff640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000efc0 tx=0x7fa8f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:44.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.397+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0009ba0 con 0x7fa8f815a420 2024-03-16T20:07:44.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.397+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f000f040 con 0x7fa8f815a420 2024-03-16T20:07:44.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.397+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00146c0 con 0x7fa8f815a420 2024-03-16T20:07:44.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.397+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f815d560 con 0x7fa8f815a420 2024-03-16T20:07:44.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.397+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f81f6160 con 0x7fa8f815a420 2024-03-16T20:07:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.401+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f815d3c0 con 0x7fa8f815a420 2024-03-16T20:07:44.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.437+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa8f0014820 con 0x7fa8f815a420 2024-03-16T20:07:44.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.437+0000 7fa8dffff640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 0x7fa8e4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:44.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.437+0000 7fa8f77fe640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 0x7fa8e4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:44.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.437+0000 7fa8f77fe640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 0x7fa8e4069380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8007fd0 tx=0x7fa8e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.721+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa8f00565b0 con 0x7fa8f815a420 2024-03-16T20:07:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:44.721+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa8f0052850 con 0x7fa8f815a420 2024-03-16T20:07:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.093+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8f815d3c0 con 0x7fa8f815a420 2024-03-16T20:07:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.113+0000 7fa8dffff640 1 -- 172.21.15.32:0/1607394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fa8f815d3c0 con 0x7fa8f815a420 2024-03-16T20:07:45.101 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:45.101 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 msgr2=0x7fa8e4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 0x7fa8e4069380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8007fd0 tx=0x7fa8e8009040 comp rx=0 tx=0).stop 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 msgr2=0x7fa8f815bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000efc0 tx=0x7fa8f000c490 comp rx=0 tx=0).stop 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 shutdown_connections 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa8e4066ec0 0x7fa8e4069380 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8f815a420 0x7fa8f815bb00 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f8150d70 0x7fa8f814b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 --2- 172.21.15.32:0/1607394 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8f80725e0 0x7fa8f81f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 >> 172.21.15.32:0/1607394 conn(0x7fa8f806c0d0 msgr2=0x7fa8f814e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 shutdown_connections 2024-03-16T20:07:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:45.117+0000 7fa8fde2f640 1 -- 172.21.15.32:0/1607394 wait complete. 2024-03-16T20:07:45.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:45.731 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:46.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.853+0000 7fe1ada46640 1 Processor -- start 2024-03-16T20:07:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.853+0000 7fe1ada46640 1 -- start start 2024-03-16T20:07:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1ada46640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe1a80725e0 0x7fe1a8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1ada46640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1a815a420 0x7fe1a815a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1ada46640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a8151020 con 0x7fe1a8150c40 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1ada46640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a815a000 con 0x7fe1a815a420 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1ada46640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a815add0 con 0x7fe1a80725e0 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1a77fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1a815a420 0x7fe1a815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1a77fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1a815a420 0x7fe1a815a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41092/0 (socket says 172.21.15.32:41092) 2024-03-16T20:07:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1a77fe640 1 -- 172.21.15.32:0/4291154977 learned_addr learned my addr 172.21.15.32:0/4291154977 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.857+0000 7fe1a67fc640 1 --2- 172.21.15.32:0/4291154977 >> v2:172.21.15.59:3300/0 conn(0x7fe1a80725e0 0x7fe1a8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.861+0000 7fe1a77fe640 1 -- 172.21.15.32:0/4291154977 >> v2:172.21.15.59:3300/0 conn(0x7fe1a80725e0 msgr2=0x7fe1a8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.861+0000 7fe1a77fe640 1 --2- 172.21.15.32:0/4291154977 >> v2:172.21.15.59:3300/0 conn(0x7fe1a80725e0 0x7fe1a8073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.861+0000 7fe1a77fe640 1 -- 172.21.15.32:0/4291154977 >> v1:172.21.15.59:6789/0 conn(0x7fe1a8150c40 legacy=0x7fe1a8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.861+0000 7fe1a77fe640 1 -- 172.21.15.32:0/4291154977 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a8072da0 con 0x7fe1a815a420 2024-03-16T20:07:46.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.865+0000 7fe1a77fe640 1 --2- 172.21.15.32:0/4291154977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a815a890 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe1a806fba0 tx=0x7fe1a001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3087eef796946f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1a5ffb640 1 -- 172.21.15.32:0/4291154977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00a58b0 con 0x7fe1a815a420 2024-03-16T20:07:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1a5ffb640 1 -- 172.21.15.32:0/4291154977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00a5a10 con 0x7fe1a815a420 2024-03-16T20:07:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1a5ffb640 1 -- 172.21.15.32:0/4291154977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00a5da0 con 0x7fe1a815a420 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 -- 172.21.15.32:0/4291154977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 msgr2=0x7fe1a815a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 --2- 172.21.15.32:0/4291154977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a815a890 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe1a806fba0 tx=0x7fe1a001c9f0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 -- 172.21.15.32:0/4291154977 shutdown_connections 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 --2- 172.21.15.32:0/4291154977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a815a890 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 --2- 172.21.15.32:0/4291154977 >> v2:172.21.15.59:3300/0 conn(0x7fe1a80725e0 0x7fe1a8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 -- 172.21.15.32:0/4291154977 >> 172.21.15.32:0/4291154977 conn(0x7fe1a806c0d0 msgr2=0x7fe1a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 -- 172.21.15.32:0/4291154977 shutdown_connections 2024-03-16T20:07:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.869+0000 7fe1ada46640 1 -- 172.21.15.32:0/4291154977 wait complete. 2024-03-16T20:07:46.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 Processor -- start 2024-03-16T20:07:46.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 -- start start 2024-03-16T20:07:46.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80725e0 0x7fe1a81f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1a8150c40 0x7fe1a814ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80c6980 con 0x7fe1a815a420 2024-03-16T20:07:46.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8144d60 con 0x7fe1a80725e0 2024-03-16T20:07:46.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1ada46640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81f6f80 con 0x7fe1a8150c40 2024-03-16T20:07:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41094/0 (socket says 172.21.15.32:41094) 2024-03-16T20:07:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1a77fe640 1 -- 172.21.15.32:0/2170169039 learned_addr learned my addr 172.21.15.32:0/2170169039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1a67fc640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1a8150c40 0x7fe1a814ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.873+0000 7fe1a6ffd640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80725e0 0x7fe1a81f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.877+0000 7fe1a77fe640 1 -- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1a8150c40 msgr2=0x7fe1a814ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.877+0000 7fe1a77fe640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1a8150c40 0x7fe1a814ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.877+0000 7fe1a77fe640 1 -- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80725e0 msgr2=0x7fe1a81f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.877+0000 7fe1a77fe640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80725e0 0x7fe1a81f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.877+0000 7fe1a77fe640 1 -- 172.21.15.32:0/2170169039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a0009b30 con 0x7fe1a815a420 2024-03-16T20:07:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.881+0000 7fe1a77fe640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00af040 tx=0x7fe1a0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:46.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.885+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00b8880 con 0x7fe1a815a420 2024-03-16T20:07:46.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.885+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00b8a00 con 0x7fe1a815a420 2024-03-16T20:07:46.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.885+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00b8db0 con 0x7fe1a815a420 2024-03-16T20:07:46.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.885+0000 7fe1ada46640 1 -- 172.21.15.32:0/2170169039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8147e30 con 0x7fe1a815a420 2024-03-16T20:07:46.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.885+0000 7fe1ada46640 1 -- 172.21.15.32:0/2170169039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a81f6240 con 0x7fe1a815a420 2024-03-16T20:07:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.889+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1800052a0 con 0x7fe1a815a420 2024-03-16T20:07:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.929+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe1a0007470 con 0x7fe1a815a420 2024-03-16T20:07:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.929+0000 7fe18ffff640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 0x7fe194069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.929+0000 7fe1a6ffd640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 0x7fe194069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:46.929+0000 7fe1a6ffd640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 0x7fe194069380 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe198001770 tx=0x7fe198009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.237+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe1a00f1920 con 0x7fe1a815a420 2024-03-16T20:07:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.237+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe1a00edbc0 con 0x7fe1a815a420 2024-03-16T20:07:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.517+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe180005900 con 0x7fe1a815a420 2024-03-16T20:07:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.541+0000 7fe18ffff640 1 -- 172.21.15.32:0/2170169039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fe180005900 con 0x7fe1a815a420 2024-03-16T20:07:47.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:47.529 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 msgr2=0x7fe194069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 0x7fe194069380 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe198001770 tx=0x7fe198009040 comp rx=0 tx=0).stop 2024-03-16T20:07:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 msgr2=0x7fe1a80c4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00af040 tx=0x7fe1a0019790 comp rx=0 tx=0).stop 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 shutdown_connections 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe194066ec0 0x7fe194069380 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1a815a420 0x7fe1a80c4b30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1a8150c40 0x7fe1a814ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 --2- 172.21.15.32:0/2170169039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80725e0 0x7fe1a81f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 >> 172.21.15.32:0/2170169039 conn(0x7fe1a806c0d0 msgr2=0x7fe1a806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 shutdown_connections 2024-03-16T20:07:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:47.545+0000 7fe18dffb640 1 -- 172.21.15.32:0/2170169039 wait complete. 2024-03-16T20:07:48.159 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:48.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:49.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:49.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.281+0000 7f58f3926640 1 Processor -- start 2024-03-16T20:07:49.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.281+0000 7f58f3926640 1 -- start start 2024-03-16T20:07:49.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f3926640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58ec151ef0 0x7f58ec0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f3926640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec158c80 con 0x7f58ec15b2a0 2024-03-16T20:07:49.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f3926640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec15bfc0 con 0x7f58ec0c97b0 2024-03-16T20:07:49.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f3926640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec0b0ad0 con 0x7f58ec151ef0 2024-03-16T20:07:49.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f1ec3640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58ec15b2a0 0x7f58ec15b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54242/0 (socket says 172.21.15.32:54242) 2024-03-16T20:07:49.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f1ec3640 1 -- 172.21.15.32:0/1162583936 learned_addr learned my addr 172.21.15.32:0/1162583936 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:49.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.285+0000 7f58f16c2640 1 --2- 172.21.15.32:0/1162583936 >> v2:172.21.15.59:3300/0 conn(0x7f58ec151ef0 0x7f58ec0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.289+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776431287 0 0) 0x7f58ec158c80 con 0x7f58ec15b2a0 2024-03-16T20:07:49.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.289+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003530 con 0x7f58ec15b2a0 2024-03-16T20:07:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.289+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400475895 0 0) 0x7f58ec15bfc0 con 0x7f58ec0c97b0 2024-03-16T20:07:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.289+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec158c80 con 0x7f58ec0c97b0 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000776868 0 0) 0x7f58d8003530 con 0x7f58ec15b2a0 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 >> v2:172.21.15.59:3300/0 conn(0x7f58ec151ef0 msgr2=0x7f58ec0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 --2- 172.21.15.32:0/1162583936 >> v2:172.21.15.59:3300/0 conn(0x7f58ec151ef0 0x7f58ec0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 >> v1:172.21.15.59:6789/0 conn(0x7f58ec0c97b0 legacy=0x7f58ec0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec157ec0 con 0x7f58ec15b2a0 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f58e4001a00 con 0x7f58ec15b2a0 2024-03-16T20:07:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 >> v1:172.21.15.32:6789/0 conn(0x7f58ec15b2a0 legacy=0x7f58ec15b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d8003b60 0x7f58d8003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 0x7f58d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58d80051e0 0x7f58d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0b0ad0 con 0x7f58d80047d0 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d800c060 con 0x7f58d80051e0 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d800c220 con 0x7f58d8003b60 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58f16c2640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d8003b60 0x7f58d8003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.293+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 0x7f58d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f0ec1640 1 -- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d8003b60 msgr2=0x7f58d8003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d8003b60 0x7f58d8003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f0ec1640 1 -- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58d80051e0 msgr2=0x7f58d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58d80051e0 0x7f58d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f0ec1640 1 -- 172.21.15.32:0/1162583936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d8003530 con 0x7f58d80047d0 2024-03-16T20:07:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.301+0000 7f58f1ec3640 1 -- 172.21.15.32:0/1162583936 reap_dead start 2024-03-16T20:07:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.305+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 0x7f58d8004c50 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f58ec1558c0 tx=0x7f58dc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e0828b5fb639aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:49.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.309+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0052d0 con 0x7f58d80047d0 2024-03-16T20:07:49.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc005450 con 0x7f58d80047d0 2024-03-16T20:07:49.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58e3fff640 1 -- 172.21.15.32:0/1162583936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc003840 con 0x7f58d80047d0 2024-03-16T20:07:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 msgr2=0x7f58d8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 0x7f58d8004c50 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f58ec1558c0 tx=0x7f58dc01c540 comp rx=0 tx=0).stop 2024-03-16T20:07:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- 172.21.15.32:0/1162583936 shutdown_connections 2024-03-16T20:07:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 --2- 172.21.15.32:0/1162583936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d80047d0 0x7f58d8004c50 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- 172.21.15.32:0/1162583936 >> 172.21.15.32:0/1162583936 conn(0x7f58ec070f90 msgr2=0x7f58ec0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:49.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- 172.21.15.32:0/1162583936 shutdown_connections 2024-03-16T20:07:49.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- 172.21.15.32:0/1162583936 wait complete. 2024-03-16T20:07:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 Processor -- start 2024-03-16T20:07:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- start start 2024-03-16T20:07:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d80047d0 0x7f58ec1e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58ec06cc80 0x7f58ec0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec1f37b0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec158ec0 con 0x7f58ec06cc80 2024-03-16T20:07:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.313+0000 7f58f3926640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec1efec0 con 0x7f58d80047d0 2024-03-16T20:07:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.317+0000 7f58f0ec1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.317+0000 7f58f0ec1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47656/0 (socket says 172.21.15.32:47656) 2024-03-16T20:07:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.317+0000 7f58f0ec1640 1 -- 172.21.15.32:0/13782456 learned_addr learned my addr 172.21.15.32:0/13782456 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.317+0000 7f58f1ec3640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58ec06cc80 0x7f58ec0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.317+0000 7f58f16c2640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d80047d0 0x7f58ec1e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.321+0000 7f58f0ec1640 1 -- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d80047d0 msgr2=0x7f58ec1e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.321+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d80047d0 0x7f58ec1e4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.321+0000 7f58f0ec1640 1 -- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58ec06cc80 msgr2=0x7f58ec0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.321+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58ec06cc80 0x7f58ec0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.321+0000 7f58f0ec1640 1 -- 172.21.15.32:0/13782456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec1f0e90 con 0x7f58ec14f3e0 2024-03-16T20:07:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.325+0000 7f58f0ec1640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f58dc005040 tx=0x7f58d8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.329+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc003e40 con 0x7f58ec14f3e0 2024-03-16T20:07:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.329+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc0052e0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.329+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec1f34d0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.329+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0069e0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.329+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec1f03b0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.333+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec068000 con 0x7f58ec14f3e0 2024-03-16T20:07:49.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.641+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58dc0b2040 con 0x7f58ec14f3e0 2024-03-16T20:07:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.641+0000 7f58e1ffb640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 0x7f58d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.641+0000 7f58f16c2640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 0x7f58d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.641+0000 7f58f16c2640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 0x7f58d0069300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f58e8003670 tx=0x7f58e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.665+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f58dc0ef210 con 0x7f58ec14f3e0 2024-03-16T20:07:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.665+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f58dc0eb4b0 con 0x7f58ec14f3e0 2024-03-16T20:07:49.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.949+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ec070a40 con 0x7f58ec14f3e0 2024-03-16T20:07:49.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58e1ffb640 1 -- 172.21.15.32:0/13782456 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f58dc0b0020 con 0x7f58ec14f3e0 2024-03-16T20:07:49.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:49.959 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:49.959 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 msgr2=0x7f58d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 0x7f58d0069300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f58e8003670 tx=0x7f58e8008040 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 msgr2=0x7f58ec06c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f58dc005040 tx=0x7f58d8005800 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 shutdown_connections 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f58d0066e40 0x7f58d0069300 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58ec06cc80 0x7f58ec0c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58ec14f3e0 0x7f58ec06c740 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 --2- 172.21.15.32:0/13782456 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58d80047d0 0x7f58ec1e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 >> 172.21.15.32:0/13782456 conn(0x7f58ec070f90 msgr2=0x7f58ec06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 shutdown_connections 2024-03-16T20:07:49.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:49.973+0000 7f58f3926640 1 -- 172.21.15.32:0/13782456 wait complete. 2024-03-16T20:07:50.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:50.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:51.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 Processor -- start 2024-03-16T20:07:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 -- start start 2024-03-16T20:07:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02b40c6420 0x7f02b40c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b4070760 con 0x7f02b40c6040 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b40731c0 con 0x7f02b40c6420 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b9836640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40734a0 con 0x7f02b4158390 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b27fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02b40c6420 0x7f02b40c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02b40c6040 0x7f02b4157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54254/0 (socket says 172.21.15.32:54254) 2024-03-16T20:07:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b2ffd640 1 -- 172.21.15.32:0/2138963420 learned_addr learned my addr 172.21.15.32:0/2138963420 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:51.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.705+0000 7f02b37fe640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b1ffb640 1 -- 172.21.15.32:0/2138963420 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392272473 0 0) 0x7f02b4070760 con 0x7f02b40c6040 2024-03-16T20:07:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b1ffb640 1 -- 172.21.15.32:0/2138963420 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029c003530 con 0x7f02b40c6040 2024-03-16T20:07:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b27fc640 1 -- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 msgr2=0x7f02b41516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b27fc640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b27fc640 1 -- 172.21.15.32:0/2138963420 >> v1:172.21.15.32:6789/0 conn(0x7f02b40c6040 legacy=0x7f02b4157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.709+0000 7f02b27fc640 1 -- 172.21.15.32:0/2138963420 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b406fe30 con 0x7f02b40c6420 2024-03-16T20:07:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.717+0000 7f02b27fc640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b40c8800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b327683cf643d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b1ffb640 1 -- 172.21.15.32:0/2138963420 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80ab670 con 0x7f02b40c6420 2024-03-16T20:07:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b1ffb640 1 -- 172.21.15.32:0/2138963420 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80ab7d0 con 0x7f02b40c6420 2024-03-16T20:07:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b1ffb640 1 -- 172.21.15.32:0/2138963420 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80abb60 con 0x7f02b40c6420 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b9836640 1 -- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 msgr2=0x7f02b40c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b9836640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b40c8800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801c470 comp rx=0 tx=0).stop 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b9836640 1 -- 172.21.15.32:0/2138963420 shutdown_connections 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b9836640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.721+0000 7f02b9836640 1 --2- 172.21.15.32:0/2138963420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b40c8800 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- 172.21.15.32:0/2138963420 >> 172.21.15.32:0/2138963420 conn(0x7f02b406c0d0 msgr2=0x7f02b406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- 172.21.15.32:0/2138963420 shutdown_connections 2024-03-16T20:07:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- 172.21.15.32:0/2138963420 wait complete. 2024-03-16T20:07:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 Processor -- start 2024-03-16T20:07:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- start start 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 0x7f02b40ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b41f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40ccaa0 con 0x7f02b40c6040 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40cace0 con 0x7f02b40c6420 2024-03-16T20:07:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b9836640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40cc1b0 con 0x7f02b4158390 2024-03-16T20:07:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.725+0000 7f02b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 0x7f02b40ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.729+0000 7f02b27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b41f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.729+0000 7f02b27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b41f4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:49192/0 (socket says 172.21.15.32:49192) 2024-03-16T20:07:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.729+0000 7f02b27fc640 1 -- 172.21.15.32:0/3433521947 learned_addr learned my addr 172.21.15.32:0/3433521947 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.729+0000 7f02b37fe640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.733+0000 7f02b2ffd640 1 -- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 msgr2=0x7f02b41f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.733+0000 7f02b2ffd640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.733+0000 7f02b2ffd640 1 -- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 msgr2=0x7f02b41f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.733+0000 7f02b2ffd640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b41f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.733+0000 7f02b2ffd640 1 -- 172.21.15.32:0/3433521947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8009420 con 0x7f02b40c6040 2024-03-16T20:07:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.737+0000 7f02b2ffd640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 0x7f02b40ccec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f02a4007fd0 tx=0x7f02a400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4005cf0 con 0x7f02b40c6040 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4005e50 con 0x7f02b40c6040 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b40ca4d0 con 0x7f02b40c6040 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a40072f0 con 0x7f02b40c6040 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b40caa00 con 0x7f02b40c6040 2024-03-16T20:07:51.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:51.741+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b40c8b40 con 0x7f02b40c6040 2024-03-16T20:07:52.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.017+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02a4007450 con 0x7f02b40c6040 2024-03-16T20:07:52.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.017+0000 7f029bfff640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 0x7f02a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:52.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.017+0000 7f02b27fc640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 0x7f02a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:52.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.021+0000 7f02b27fc640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 0x7f02a0069350 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f02a801ab20 tx=0x7f02a80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.045+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f02a4055020 con 0x7f02b40c6040 2024-03-16T20:07:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.045+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f02a40512c0 con 0x7f02b40c6040 2024-03-16T20:07:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.249+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02b4068000 con 0x7f02b40c6040 2024-03-16T20:07:52.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.269+0000 7f029bfff640 1 -- 172.21.15.32:0/3433521947 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f02a4017070 con 0x7f02b40c6040 2024-03-16T20:07:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:52.258 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:52.259 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:52.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 msgr2=0x7f02a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:52.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 0x7f02a0069350 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f02a801ab20 tx=0x7f02a80a7000 comp rx=0 tx=0).stop 2024-03-16T20:07:52.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 msgr2=0x7f02b40ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:52.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 0x7f02b40ccec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f02a4007fd0 tx=0x7f02a400c890 comp rx=0 tx=0).stop 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 shutdown_connections 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f02a0066e90 0x7f02a0069350 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02b4158390 0x7f02b41f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02b40c6420 0x7f02b41f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 --2- 172.21.15.32:0/3433521947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b40c6040 0x7f02b40ccec0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 >> 172.21.15.32:0/3433521947 conn(0x7f02b406c0d0 msgr2=0x7f02b4156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 shutdown_connections 2024-03-16T20:07:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:52.273+0000 7f02b9836640 1 -- 172.21.15.32:0/3433521947 wait complete. 2024-03-16T20:07:52.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:52.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:53.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:53.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 Processor -- start 2024-03-16T20:07:53.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 -- start start 2024-03-16T20:07:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6764070840 0x7f67640c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67640724a0 0x7f67640728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6764070c90 con 0x7f6764153fa0 2024-03-16T20:07:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67640c3f00 con 0x7f6764070840 2024-03-16T20:07:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f676ac5f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6764070650 con 0x7f67640724a0 2024-03-16T20:07:53.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f6763fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6764070840 0x7f67640c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:53.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f6763fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6764070840 0x7f67640c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47668/0 (socket says 172.21.15.32:47668) 2024-03-16T20:07:53.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.005+0000 7f6763fff640 1 -- 172.21.15.32:0/3120875781 learned_addr learned my addr 172.21.15.32:0/3120875781 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:53.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.009+0000 7f67691fc640 1 --2- 172.21.15.32:0/3120875781 >> v2:172.21.15.59:3300/0 conn(0x7f67640724a0 0x7f67640728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:53.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.009+0000 7f6763fff640 1 -- 172.21.15.32:0/3120875781 >> v2:172.21.15.59:3300/0 conn(0x7f67640724a0 msgr2=0x7f67640728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:53.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.009+0000 7f6763fff640 1 --2- 172.21.15.32:0/3120875781 >> v2:172.21.15.59:3300/0 conn(0x7f67640724a0 0x7f67640728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:53.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.009+0000 7f6763fff640 1 -- 172.21.15.32:0/3120875781 >> v1:172.21.15.59:6789/0 conn(0x7f6764153fa0 legacy=0x7f676415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:53.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.009+0000 7f6763fff640 1 -- 172.21.15.32:0/3120875781 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640c8590 con 0x7f6764070840 2024-03-16T20:07:53.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.013+0000 7f6763fff640 1 --2- 172.21.15.32:0/3120875781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f67640c7470 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f67640cd770 tx=0x7f675001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c402cf581cccb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:54.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f67637fe640 1 -- 172.21.15.32:0/3120875781 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500a8a10 con 0x7f6764070840 2024-03-16T20:07:54.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f67637fe640 1 -- 172.21.15.32:0/3120875781 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67500a8b70 con 0x7f6764070840 2024-03-16T20:07:54.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f67637fe640 1 -- 172.21.15.32:0/3120875781 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500b0440 con 0x7f6764070840 2024-03-16T20:07:54.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 -- 172.21.15.32:0/3120875781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 msgr2=0x7f67640c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:54.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 --2- 172.21.15.32:0/3120875781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f67640c7470 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f67640cd770 tx=0x7f675001ed10 comp rx=0 tx=0).stop 2024-03-16T20:07:54.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 -- 172.21.15.32:0/3120875781 shutdown_connections 2024-03-16T20:07:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 --2- 172.21.15.32:0/3120875781 >> v2:172.21.15.59:3300/0 conn(0x7f67640724a0 0x7f67640728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 --2- 172.21.15.32:0/3120875781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f67640c7470 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 -- 172.21.15.32:0/3120875781 >> 172.21.15.32:0/3120875781 conn(0x7f676406c0d0 msgr2=0x7f676406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 -- 172.21.15.32:0/3120875781 shutdown_connections 2024-03-16T20:07:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.017+0000 7f676ac5f640 1 -- 172.21.15.32:0/3120875781 wait complete. 2024-03-16T20:07:54.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 Processor -- start 2024-03-16T20:07:54.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 -- start start 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67640724a0 0x7f67640ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6764153fa0 0x7f6764146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764150070 con 0x7f6764070840 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764150330 con 0x7f67640724a0 2024-03-16T20:07:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f676ac5f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6764144d40 con 0x7f6764153fa0 2024-03-16T20:07:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f67689fb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f67691fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6764153fa0 0x7f6764146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f67689fb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47670/0 (socket says 172.21.15.32:47670) 2024-03-16T20:07:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f67689fb640 1 -- 172.21.15.32:0/2769129539 learned_addr learned my addr 172.21.15.32:0/2769129539 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.021+0000 7f6763fff640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67640724a0 0x7f67640ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.029+0000 7f67689fb640 1 -- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6764153fa0 msgr2=0x7f6764146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.029+0000 7f67689fb640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6764153fa0 0x7f6764146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.029+0000 7f67689fb640 1 -- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67640724a0 msgr2=0x7f67640ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.029+0000 7f67689fb640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67640724a0 0x7f67640ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.029+0000 7f67689fb640 1 -- 172.21.15.32:0/2769129539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500094d0 con 0x7f6764070840 2024-03-16T20:07:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.033+0000 7f67689fb640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f675800c370 tx=0x7f675800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:54.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67580052d0 con 0x7f6764070840 2024-03-16T20:07:54.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6758005430 con 0x7f6764070840 2024-03-16T20:07:54.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6764144fd0 con 0x7f6764070840 2024-03-16T20:07:54.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758007240 con 0x7f6764070840 2024-03-16T20:07:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67641505a0 con 0x7f6764070840 2024-03-16T20:07:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.037+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6764068000 con 0x7f6764070840 2024-03-16T20:07:54.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.313+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f67580073a0 con 0x7f6764070840 2024-03-16T20:07:54.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.313+0000 7f67617fa640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 0x7f6748069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:54.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.313+0000 7f6763fff640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 0x7f6748069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.313+0000 7f6763fff640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 0x7f6748069380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f67500b2000 tx=0x7f6750009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:54.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.341+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6758007650 con 0x7f6764070840 2024-03-16T20:07:54.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.341+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6758050f50 con 0x7f6764070840 2024-03-16T20:07:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.557+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6764070c90 con 0x7f6764070840 2024-03-16T20:07:54.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f67617fa640 1 -- 172.21.15.32:0/2769129539 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f6764070c90 con 0x7f6764070840 2024-03-16T20:07:54.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:54.559 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:54.559 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:54.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 msgr2=0x7f6748069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 0x7f6748069380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f67500b2000 tx=0x7f6750009040 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 msgr2=0x7f6764144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f675800c370 tx=0x7f675800c830 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 shutdown_connections 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6748066ec0 0x7f6748069380 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6764153fa0 0x7f6764146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67640724a0 0x7f67640ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 --2- 172.21.15.32:0/2769129539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6764070840 0x7f6764144800 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 >> 172.21.15.32:0/2769129539 conn(0x7f676406c0d0 msgr2=0x7f6764149a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 shutdown_connections 2024-03-16T20:07:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:54.573+0000 7f676ac5f640 1 -- 172.21.15.32:0/2769129539 wait complete. 2024-03-16T20:07:55.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:55.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:56.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 Processor -- start 2024-03-16T20:07:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 -- start start 2024-03-16T20:07:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec840725e0 0x7fec84073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fec8415a420 0x7fec8415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec84072dd0 con 0x7fec84150d70 2024-03-16T20:07:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8415d790 con 0x7fec840725e0 2024-03-16T20:07:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.301+0000 7fec8c0d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8415da50 con 0x7fec8415a420 2024-03-16T20:07:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.305+0000 7fec89672640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec840725e0 0x7fec84073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.305+0000 7fec8a674640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fec8415a420 0x7fec8415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.305+0000 7fec89672640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec840725e0 0x7fec84073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47676/0 (socket says 172.21.15.32:47676) 2024-03-16T20:07:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.305+0000 7fec89672640 1 -- 172.21.15.32:0/1290114620 learned_addr learned my addr 172.21.15.32:0/1290114620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:56.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.309+0000 7fec89672640 1 -- 172.21.15.32:0/1290114620 >> v2:172.21.15.59:3300/0 conn(0x7fec8415a420 msgr2=0x7fec8415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:56.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.309+0000 7fec89672640 1 --2- 172.21.15.32:0/1290114620 >> v2:172.21.15.59:3300/0 conn(0x7fec8415a420 0x7fec8415a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.309+0000 7fec89672640 1 -- 172.21.15.32:0/1290114620 >> v1:172.21.15.32:6789/0 conn(0x7fec84150d70 legacy=0x7fec84070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.309+0000 7fec89672640 1 -- 172.21.15.32:0/1290114620 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8415dd30 con 0x7fec840725e0 2024-03-16T20:07:56.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.313+0000 7fec89672640 1 --2- 172.21.15.32:0/1290114620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec84073070 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fec840c4a30 tx=0x7fec7401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876e6df128307120 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:56.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.317+0000 7fec88e71640 1 -- 172.21.15.32:0/1290114620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740ac630 con 0x7fec840725e0 2024-03-16T20:07:56.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.317+0000 7fec88e71640 1 -- 172.21.15.32:0/1290114620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec740ac790 con 0x7fec840725e0 2024-03-16T20:07:56.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.317+0000 7fec88e71640 1 -- 172.21.15.32:0/1290114620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740acb20 con 0x7fec840725e0 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- 172.21.15.32:0/1290114620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 msgr2=0x7fec84073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/1290114620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec84073070 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fec840c4a30 tx=0x7fec7401c3d0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- 172.21.15.32:0/1290114620 shutdown_connections 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/1290114620 >> v2:172.21.15.59:3300/0 conn(0x7fec8415a420 0x7fec8415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/1290114620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec84073070 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- 172.21.15.32:0/1290114620 >> 172.21.15.32:0/1290114620 conn(0x7fec8406c0d0 msgr2=0x7fec8406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:56.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- 172.21.15.32:0/1290114620 shutdown_connections 2024-03-16T20:07:56.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- 172.21.15.32:0/1290114620 wait complete. 2024-03-16T20:07:56.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 Processor -- start 2024-03-16T20:07:56.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- start start 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec84150d70 0x7fec84141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec8415a420 0x7fec8415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec841f6210 con 0x7fec840725e0 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec841f7130 con 0x7fec84150d70 2024-03-16T20:07:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.321+0000 7fec8c0d7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8415caf0 con 0x7fec8415a420 2024-03-16T20:07:56.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.325+0000 7fec89e73640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.325+0000 7fec89672640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec84150d70 0x7fec84141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.325+0000 7fec89e73640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47686/0 (socket says 172.21.15.32:47686) 2024-03-16T20:07:56.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.325+0000 7fec89e73640 1 -- 172.21.15.32:0/3821539549 learned_addr learned my addr 172.21.15.32:0/3821539549 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:56.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.325+0000 7fec8a674640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec8415a420 0x7fec8415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.329+0000 7fec89e73640 1 -- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec8415a420 msgr2=0x7fec8415bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.329+0000 7fec89e73640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec8415a420 0x7fec8415bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.329+0000 7fec89e73640 1 -- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec84150d70 msgr2=0x7fec84141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.329+0000 7fec89e73640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec84150d70 0x7fec84141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.329+0000 7fec89e73640 1 -- 172.21.15.32:0/3821539549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740094d0 con 0x7fec840725e0 2024-03-16T20:07:56.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.333+0000 7fec89e73640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c3d0 tx=0x7fec8000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.337+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec800052d0 con 0x7fec840725e0 2024-03-16T20:07:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.337+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec80005430 con 0x7fec840725e0 2024-03-16T20:07:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.337+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8415d560 con 0x7fec840725e0 2024-03-16T20:07:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.337+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80007260 con 0x7fec840725e0 2024-03-16T20:07:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.337+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec841f6760 con 0x7fec840725e0 2024-03-16T20:07:56.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.341+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec84068000 con 0x7fec840725e0 2024-03-16T20:07:56.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.365+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec800073c0 con 0x7fec840725e0 2024-03-16T20:07:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.365+0000 7fec7a7fc640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 0x7fec68069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.365+0000 7fec89672640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 0x7fec68069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.365+0000 7fec89672640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 0x7fec68069350 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fec740ae000 tx=0x7fec74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:56.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.641+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fec80007670 con 0x7fec840725e0 2024-03-16T20:07:56.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.641+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fec80051100 con 0x7fec840725e0 2024-03-16T20:07:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.849+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec8406f390 con 0x7fec840725e0 2024-03-16T20:07:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.873+0000 7fec7a7fc640 1 -- 172.21.15.32:0/3821539549 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7fec8406f390 con 0x7fec840725e0 2024-03-16T20:07:56.860 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:56.860 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 msgr2=0x7fec68069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 0x7fec68069350 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fec740ae000 tx=0x7fec74008040 comp rx=0 tx=0).stop 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 msgr2=0x7fec841f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c3d0 tx=0x7fec8000c890 comp rx=0 tx=0).stop 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 shutdown_connections 2024-03-16T20:07:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fec68066e90 0x7fec68069350 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec8415a420 0x7fec8415bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec84150d70 0x7fec84141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 --2- 172.21.15.32:0/3821539549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec840725e0 0x7fec841f6bf0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 >> 172.21.15.32:0/3821539549 conn(0x7fec8406c0d0 msgr2=0x7fec84147700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 shutdown_connections 2024-03-16T20:07:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:56.877+0000 7fec8c0d7640 1 -- 172.21.15.32:0/3821539549 wait complete. 2024-03-16T20:07:57.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:57.485 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:07:58.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:07:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 Processor -- start 2024-03-16T20:07:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 -- start start 2024-03-16T20:07:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34f80c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80c8ec0 con 0x7f34f80c36b0 2024-03-16T20:07:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8072950 con 0x7f34f80c8980 2024-03-16T20:07:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.609+0000 7f34ffded640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8072f10 con 0x7f34f8151000 2024-03-16T20:07:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.613+0000 7f34fd388640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f34f80c36b0 0x7f34f80c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54264/0 (socket says 172.21.15.32:54264) 2024-03-16T20:07:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.613+0000 7f34fd388640 1 -- 172.21.15.32:0/2119006828 learned_addr learned my addr 172.21.15.32:0/2119006828 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:07:58.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.613+0000 7f34fdb89640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34f80c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.613+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011066049 0 0) 0x7f34f80c8ec0 con 0x7f34f80c36b0 2024-03-16T20:07:58.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.613+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e4003530 con 0x7f34f80c36b0 2024-03-16T20:07:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239697236 0 0) 0x7f34e4003530 con 0x7f34f80c36b0 2024-03-16T20:07:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 msgr2=0x7f34f80c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34f80c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 >> v1:172.21.15.59:6789/0 conn(0x7f34f80c8980 legacy=0x7f34f81615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8072c30 con 0x7f34f80c36b0 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f34e8001680 con 0x7f34f80c36b0 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 >> v1:172.21.15.32:6789/0 conn(0x7f34f80c36b0 legacy=0x7f34f80c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e40039b0 0x7f34e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34e40047d0 0x7f34e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34e4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8072f10 con 0x7f34e40039b0 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e400ba40 con 0x7f34e40047d0 2024-03-16T20:07:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.617+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e400bbb0 con 0x7f34f8151000 2024-03-16T20:07:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.621+0000 7f34fdb89640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e40039b0 0x7f34e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.621+0000 7f34fd388640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34e40047d0 0x7f34e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fe38a640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34e4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fdb89640 1 -- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 msgr2=0x7f34e4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:58.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fdb89640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8151000 0x7f34e4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fdb89640 1 -- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34e40047d0 msgr2=0x7f34e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:07:58.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fdb89640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34e40047d0 0x7f34e4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.625+0000 7f34fdb89640 1 -- 172.21.15.32:0/2119006828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4003530 con 0x7f34e40039b0 2024-03-16T20:07:58.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.633+0000 7f34fdb89640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e40039b0 0x7f34e4003d90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009ab0 tx=0x7f34f401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0599bfd34bd4665 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40a6500 con 0x7f34e40039b0 2024-03-16T20:07:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f40a6660 con 0x7f34e40039b0 2024-03-16T20:07:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34fcb87640 1 -- 172.21.15.32:0/2119006828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40a6a10 con 0x7f34e40039b0 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 -- 172.21.15.32:0/2119006828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e40039b0 msgr2=0x7f34e4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 --2- 172.21.15.32:0/2119006828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e40039b0 0x7f34e4003d90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009ab0 tx=0x7f34f401c9d0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34fe38a640 1 -- 172.21.15.32:0/2119006828 reap_dead start 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 -- 172.21.15.32:0/2119006828 shutdown_connections 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 -- 172.21.15.32:0/2119006828 >> 172.21.15.32:0/2119006828 conn(0x7f34f806c0d0 msgr2=0x7f34f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 -- 172.21.15.32:0/2119006828 shutdown_connections 2024-03-16T20:07:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.637+0000 7f34ffded640 1 -- 172.21.15.32:0/2119006828 wait complete. 2024-03-16T20:07:58.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 Processor -- start 2024-03-16T20:07:58.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 -- start start 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34f81422a0 0x7f34f8157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8158370 0x7f34f813fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f814cda0 con 0x7f34f8141ec0 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f81f55a0 con 0x7f34f81422a0 2024-03-16T20:07:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34ffded640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f814f700 con 0x7f34f8158370 2024-03-16T20:07:58.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34fdb89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34fdb89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47718/0 (socket says 172.21.15.32:47718) 2024-03-16T20:07:58.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34fdb89640 1 -- 172.21.15.32:0/962850364 learned_addr learned my addr 172.21.15.32:0/962850364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:07:58.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34fd388640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34f81422a0 0x7f34f8157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.641+0000 7f34fe38a640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8158370 0x7f34f813fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 -- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8158370 msgr2=0x7f34f813fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:58.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8158370 0x7f34f813fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 -- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34f81422a0 msgr2=0x7f34f8157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:58.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34f81422a0 0x7f34f8157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:58.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 -- 172.21.15.32:0/962850364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f814cf90 con 0x7f34f8141ec0 2024-03-16T20:07:58.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.649+0000 7f34fdb89640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f34f401c800 tx=0x7f34e4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40be070 con 0x7f34f8141ec0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400f0c0 con 0x7f34f8141ec0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34ffded640 1 -- 172.21.15.32:0/962850364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f814d250 con 0x7f34f8141ec0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40a6a10 con 0x7f34f8141ec0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34ffded640 1 -- 172.21.15.32:0/962850364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f814e940 con 0x7f34f8141ec0 2024-03-16T20:07:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.657+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d40052a0 con 0x7f34f8141ec0 2024-03-16T20:07:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.937+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f34f40a6b70 con 0x7f34f8141ec0 2024-03-16T20:07:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.941+0000 7f34ee7fc640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 0x7f34dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:07:58.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.941+0000 7f34fd388640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 0x7f34dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:07:58.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.941+0000 7f34fd388640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 0x7f34dc069350 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f34f8145b80 tx=0x7f34e8000f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:07:58.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.965+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f34f40aa070 con 0x7f34f8141ec0 2024-03-16T20:07:58.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:58.965+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f34f40ebf90 con 0x7f34f8141ec0 2024-03-16T20:07:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.181+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34d4005900 con 0x7f34f8141ec0 2024-03-16T20:07:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.193+0000 7f34ee7fc640 1 -- 172.21.15.32:0/962850364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6038 (secure 0 0 0) 0x7f34d4005900 con 0x7f34f8141ec0 2024-03-16T20:07:59.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:07:59.182 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:04:18.601918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":35,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"laggy_since":"2024-03-16T20:04:18.601913+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4242,4255,4243,4252]},"id":1}]} 2024-03-16T20:07:59.183 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-03-16T20:07:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 msgr2=0x7f34dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 0x7f34dc069350 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f34f8145b80 tx=0x7f34e8000f90 comp rx=0 tx=0).stop 2024-03-16T20:07:59.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 msgr2=0x7f34f81f5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f34f401c800 tx=0x7f34e4005800 comp rx=0 tx=0).stop 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 shutdown_connections 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f34dc066e90 0x7f34dc069350 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34f8158370 0x7f34f813fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34f81422a0 0x7f34f8157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 --2- 172.21.15.32:0/962850364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34f8141ec0 0x7f34f81f5060 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 >> 172.21.15.32:0/962850364 conn(0x7f34f806c0d0 msgr2=0x7f34f8156e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 shutdown_connections 2024-03-16T20:07:59.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:07:59.197+0000 7f34dbfff640 1 -- 172.21.15.32:0/962850364 wait complete. 2024-03-16T20:07:59.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:04:18.601918+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T20:04:18.601913+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4242, 4255, 4243, 4252]} 2024-03-16T20:07:59.805 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:00.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:00.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 Processor -- start 2024-03-16T20:08:00.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 -- start start 2024-03-16T20:08:00.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6040 0x7f05f00c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05f00c6420 0x7f05f0157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f0070760 con 0x7f05f006ba30 2024-03-16T20:08:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00731c0 con 0x7f05f00c6420 2024-03-16T20:08:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f70c6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00734a0 con 0x7f05f00c6040 2024-03-16T20:08:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05effff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05f00c6420 0x7f05f0157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05effff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05f00c6420 0x7f05f0157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56088/0 (socket says 172.21.15.32:56088) 2024-03-16T20:08:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05effff640 1 -- 172.21.15.32:0/1401274217 learned_addr learned my addr 172.21.15.32:0/1401274217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:00.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.925+0000 7f05f4e62640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6040 0x7f05f00c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.929+0000 7f05effff640 1 -- 172.21.15.32:0/1401274217 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6040 msgr2=0x7f05f00c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.929+0000 7f05effff640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6040 0x7f05f00c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.929+0000 7f05effff640 1 -- 172.21.15.32:0/1401274217 >> v1:172.21.15.59:6789/0 conn(0x7f05f006ba30 legacy=0x7f05f01516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.929+0000 7f05effff640 1 -- 172.21.15.32:0/1401274217 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f006fe30 con 0x7f05f00c6420 2024-03-16T20:08:00.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.933+0000 7f05effff640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f00c6420 0x7f05f0157c80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009870 tx=0x7f05dc01c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac9a6e06eb0821e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:00.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05ef7fe640 1 -- 172.21.15.32:0/1401274217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0ab660 con 0x7f05f00c6420 2024-03-16T20:08:00.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05ef7fe640 1 -- 172.21.15.32:0/1401274217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc0ab7c0 con 0x7f05f00c6420 2024-03-16T20:08:00.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05ef7fe640 1 -- 172.21.15.32:0/1401274217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0abb90 con 0x7f05f00c6420 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 -- 172.21.15.32:0/1401274217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f00c6420 msgr2=0x7f05f0157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f00c6420 0x7f05f0157c80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009870 tx=0x7f05dc01c550 comp rx=0 tx=0).stop 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 -- 172.21.15.32:0/1401274217 shutdown_connections 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f00c6420 0x7f05f0157c80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 --2- 172.21.15.32:0/1401274217 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6040 0x7f05f00c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 -- 172.21.15.32:0/1401274217 >> 172.21.15.32:0/1401274217 conn(0x7f05f006c0d0 msgr2=0x7f05f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 -- 172.21.15.32:0/1401274217 shutdown_connections 2024-03-16T20:08:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.937+0000 7f05f70c6640 1 -- 172.21.15.32:0/1401274217 wait complete. 2024-03-16T20:08:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 Processor -- start 2024-03-16T20:08:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 -- start start 2024-03-16T20:08:00.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05f00c6040 0x7f05f0148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6420 0x7f05f0149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f01f1fb0 con 0x7f05f006ba30 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f01f2290 con 0x7f05f00c6040 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f70c6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f01f2570 con 0x7f05f00c6420 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f4e62640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f4e62640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56102/0 (socket says 172.21.15.32:56102) 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.941+0000 7f05f4e62640 1 -- 172.21.15.32:0/3524886962 learned_addr learned my addr 172.21.15.32:0/3524886962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05effff640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05f00c6040 0x7f05f0148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f5663640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6420 0x7f05f0149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f4e62640 1 -- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6420 msgr2=0x7f05f0149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f4e62640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6420 0x7f05f0149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f4e62640 1 -- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05f00c6040 msgr2=0x7f05f0148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f4e62640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05f00c6040 0x7f05f0148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.945+0000 7f05f4e62640 1 -- 172.21.15.32:0/3524886962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc0094d0 con 0x7f05f006ba30 2024-03-16T20:08:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.949+0000 7f05f4e62640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05e400e9b0 tx=0x7f05e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.953+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40096b0 con 0x7f05f006ba30 2024-03-16T20:08:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.953+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e400c9a0 con 0x7f05f006ba30 2024-03-16T20:08:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.953+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f01f2850 con 0x7f05f006ba30 2024-03-16T20:08:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.953+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e400cd30 con 0x7f05f006ba30 2024-03-16T20:08:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.953+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01f2e10 con 0x7f05f006ba30 2024-03-16T20:08:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.957+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f01f3020 con 0x7f05f006ba30 2024-03-16T20:08:00.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.981+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f05e4013470 con 0x7f05f006ba30 2024-03-16T20:08:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.981+0000 7f05ed7fa640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 0x7f05d4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.981+0000 7f05effff640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 0x7f05d4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:00.981+0000 7f05effff640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 0x7f05d4069330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0af000 tx=0x7f05dc0b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.321+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f05e4056030 con 0x7f05f006ba30 2024-03-16T20:08:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.321+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f05e40522d0 con 0x7f05f006ba30 2024-03-16T20:08:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.541+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05f006fa50 con 0x7f05f006ba30 2024-03-16T20:08:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.561+0000 7f05ed7fa640 1 -- 172.21.15.32:0/3524886962 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f05e4021090 con 0x7f05f006ba30 2024-03-16T20:08:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:01.549 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 msgr2=0x7f05d4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 0x7f05d4069330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0af000 tx=0x7f05dc0b6040 comp rx=0 tx=0).stop 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 msgr2=0x7f05f01465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05e400e9b0 tx=0x7f05e400ee70 comp rx=0 tx=0).stop 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 shutdown_connections 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f05d4066e70 0x7f05d4069330 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f05f00c6420 0x7f05f0149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05f00c6040 0x7f05f0148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 --2- 172.21.15.32:0/3524886962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05f006ba30 0x7f05f01465c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 >> 172.21.15.32:0/3524886962 conn(0x7f05f006c0d0 msgr2=0x7f05f01558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 shutdown_connections 2024-03-16T20:08:01.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:01.565+0000 7f05f70c6640 1 -- 172.21.15.32:0/3524886962 wait complete. 2024-03-16T20:08:02.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:02.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:03.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:03.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 Processor -- start 2024-03-16T20:08:03.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 -- start start 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8fc8151ef0 0x7f8fc80c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fc80c97b0 0x7f8fc80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc80c8720 0x7f8fc815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc80b0ad0 con 0x7f8fc8151ef0 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc8157cb0 con 0x7f8fc80c97b0 2024-03-16T20:08:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.292+0000 7f8fd0094640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8158c80 con 0x7f8fc80c8720 2024-03-16T20:08:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8fc8151ef0 0x7f8fc80c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fce631640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc80c8720 0x7f8fc815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcd62f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fc80c97b0 0x7f8fc80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8fc8151ef0 0x7f8fc80c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56106/0 (socket says 172.21.15.32:56106) 2024-03-16T20:08:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 -- 172.21.15.32:0/2442660593 learned_addr learned my addr 172.21.15.32:0/2442660593 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 -- 172.21.15.32:0/2442660593 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc80c8720 msgr2=0x7f8fc815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2442660593 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc80c8720 0x7f8fc815b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 -- 172.21.15.32:0/2442660593 >> v2:172.21.15.59:3300/0 conn(0x7f8fc80c97b0 msgr2=0x7f8fc80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2442660593 >> v2:172.21.15.59:3300/0 conn(0x7f8fc80c97b0 0x7f8fc80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.296+0000 7f8fcde30640 1 -- 172.21.15.32:0/2442660593 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc815bfc0 con 0x7f8fc8151ef0 2024-03-16T20:08:03.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.300+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2442660593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc8151ef0 0x7f8fc80c9270 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80cad10 tx=0x7f8fc401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43e7fbd9572b1f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:03.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.304+0000 7f8fcce2e640 1 -- 172.21.15.32:0/2442660593 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40a89d0 con 0x7f8fc8151ef0 2024-03-16T20:08:03.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fcce2e640 1 -- 172.21.15.32:0/2442660593 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40a8b30 con 0x7f8fc8151ef0 2024-03-16T20:08:03.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fcce2e640 1 -- 172.21.15.32:0/2442660593 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40b0440 con 0x7f8fc8151ef0 2024-03-16T20:08:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 -- 172.21.15.32:0/2442660593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc8151ef0 msgr2=0x7f8fc80c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2442660593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc8151ef0 0x7f8fc80c9270 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80cad10 tx=0x7f8fc401ed10 comp rx=0 tx=0).stop 2024-03-16T20:08:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 -- 172.21.15.32:0/2442660593 shutdown_connections 2024-03-16T20:08:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2442660593 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc80c8720 0x7f8fc815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2442660593 >> v2:172.21.15.59:3300/0 conn(0x7f8fc80c97b0 0x7f8fc80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2442660593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc8151ef0 0x7f8fc80c9270 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 -- 172.21.15.32:0/2442660593 >> 172.21.15.32:0/2442660593 conn(0x7f8fc8070f90 msgr2=0x7f8fc80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 -- 172.21.15.32:0/2442660593 shutdown_connections 2024-03-16T20:08:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.308+0000 7f8fd0094640 1 -- 172.21.15.32:0/2442660593 wait complete. 2024-03-16T20:08:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 Processor -- start 2024-03-16T20:08:03.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 -- start start 2024-03-16T20:08:03.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fc80c97b0 0x7f8fc8158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc8151ef0 0x7f8fc80c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc814baf0 con 0x7f8fc80c8720 2024-03-16T20:08:03.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc814e140 con 0x7f8fc80c97b0 2024-03-16T20:08:03.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fd0094640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc814c2d0 con 0x7f8fc8151ef0 2024-03-16T20:08:03.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fcde30640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fcde30640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56114/0 (socket says 172.21.15.32:56114) 2024-03-16T20:08:03.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fcde30640 1 -- 172.21.15.32:0/2309011756 learned_addr learned my addr 172.21.15.32:0/2309011756 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:03.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.312+0000 7f8fce631640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc8151ef0 0x7f8fc80c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcd62f640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fc80c97b0 0x7f8fc8158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcde30640 1 -- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc8151ef0 msgr2=0x7f8fc80c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc8151ef0 0x7f8fc80c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcde30640 1 -- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fc80c97b0 msgr2=0x7f8fc8158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fc80c97b0 0x7f8fc8158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.316+0000 7f8fcde30640 1 -- 172.21.15.32:0/2309011756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc40094d0 con 0x7f8fc80c8720 2024-03-16T20:08:03.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.324+0000 7f8fcde30640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4000c10 tx=0x7f8fc401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40b0d40 con 0x7f8fc80c8720 2024-03-16T20:08:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40bb460 con 0x7f8fc80c8720 2024-03-16T20:08:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc814de60 con 0x7f8fc80c8720 2024-03-16T20:08:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40b0ea0 con 0x7f8fc80c8720 2024-03-16T20:08:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc814c850 con 0x7f8fc80c8720 2024-03-16T20:08:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.328+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8068000 con 0x7f8fc80c8720 2024-03-16T20:08:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.600+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8fc40b83f0 con 0x7f8fc80c8720 2024-03-16T20:08:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.604+0000 7f8fbe7fc640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 0x7f8fac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.604+0000 7f8fcd62f640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 0x7f8fac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.604+0000 7f8fcd62f640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 0x7f8fac069300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ca7a0 tx=0x7f8fb8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:03.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.624+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8fc40f3260 con 0x7f8fc80c8720 2024-03-16T20:08:03.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.624+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8fc40ef500 con 0x7f8fc80c8720 2024-03-16T20:08:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.828+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc815bab0 con 0x7f8fc80c8720 2024-03-16T20:08:03.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.916+0000 7f8fbe7fc640 1 -- 172.21.15.32:0/2309011756 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8fc40c0030 con 0x7f8fc80c8720 2024-03-16T20:08:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:03.904 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:03.904 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 msgr2=0x7f8fac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 0x7f8fac069300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ca7a0 tx=0x7f8fb8009190 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 msgr2=0x7f8fc81f0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4000c10 tx=0x7f8fc401bab0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 shutdown_connections 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8fac066e40 0x7f8fac069300 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8fc8151ef0 0x7f8fc80c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fc80c97b0 0x7f8fc8158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 --2- 172.21.15.32:0/2309011756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fc80c8720 0x7f8fc81f0940 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 >> 172.21.15.32:0/2309011756 conn(0x7f8fc8070f90 msgr2=0x7f8fc80c3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 shutdown_connections 2024-03-16T20:08:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:03.920+0000 7f8fd0094640 1 -- 172.21.15.32:0/2309011756 wait complete. 2024-03-16T20:08:04.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:04.528 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:05.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:05.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 Processor -- start 2024-03-16T20:08:05.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 -- start start 2024-03-16T20:08:05.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d41540c0 0x7fe5d415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:05.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5d415ac10 0x7fe5d40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d406c460 con 0x7fe5d40c8720 2024-03-16T20:08:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d415a240 con 0x7fe5d415ac10 2024-03-16T20:08:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5da5b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d406cf20 con 0x7fe5d41540c0 2024-03-16T20:08:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5d37fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5d415ac10 0x7fe5d40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5d8b50640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5d40c8720 0x7fe5d40c3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:45386/0 (socket says 172.21.15.32:45386) 2024-03-16T20:08:05.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5d8b50640 1 -- 172.21.15.32:0/1846834517 learned_addr learned my addr 172.21.15.32:0/1846834517 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:05.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.664+0000 7fe5d3fff640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d41540c0 0x7fe5d415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.668+0000 7fe5d37fe640 1 -- 172.21.15.32:0/1846834517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d41540c0 msgr2=0x7fe5d415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.668+0000 7fe5d37fe640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d41540c0 0x7fe5d415a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.668+0000 7fe5d37fe640 1 -- 172.21.15.32:0/1846834517 >> v1:172.21.15.59:6789/0 conn(0x7fe5d40c8720 legacy=0x7fe5d40c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.668+0000 7fe5d37fe640 1 -- 172.21.15.32:0/1846834517 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d406d200 con 0x7fe5d415ac10 2024-03-16T20:08:05.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.672+0000 7fe5d37fe640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d415ac10 0x7fe5d40c81e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80097c0 tx=0x7fe5c801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f56a298243fcdda server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5d2ffd640 1 -- 172.21.15.32:0/1846834517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80aa5f0 con 0x7fe5d415ac10 2024-03-16T20:08:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5d2ffd640 1 -- 172.21.15.32:0/1846834517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c80aa750 con 0x7fe5d415ac10 2024-03-16T20:08:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5d2ffd640 1 -- 172.21.15.32:0/1846834517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80aaae0 con 0x7fe5d415ac10 2024-03-16T20:08:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 -- 172.21.15.32:0/1846834517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d415ac10 msgr2=0x7fe5d40c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d415ac10 0x7fe5d40c81e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80097c0 tx=0x7fe5c801c470 comp rx=0 tx=0).stop 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 -- 172.21.15.32:0/1846834517 shutdown_connections 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d415ac10 0x7fe5d40c81e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/1846834517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d41540c0 0x7fe5d415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 -- 172.21.15.32:0/1846834517 >> 172.21.15.32:0/1846834517 conn(0x7fe5d4070f90 msgr2=0x7fe5d40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 -- 172.21.15.32:0/1846834517 shutdown_connections 2024-03-16T20:08:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.676+0000 7fe5da5b3640 1 -- 172.21.15.32:0/1846834517 wait complete. 2024-03-16T20:08:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 Processor -- start 2024-03-16T20:08:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 -- start start 2024-03-16T20:08:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d41540c0 0x7fe5d40cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d415ac10 0x7fe5d41f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:05.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d414bd20 con 0x7fe5d40c8720 2024-03-16T20:08:05.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d414c250 con 0x7fe5d41540c0 2024-03-16T20:08:05.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5da5b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d414bf70 con 0x7fe5d415ac10 2024-03-16T20:08:05.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5d3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:05.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d41540c0 0x7fe5d40cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:05.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5d3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56136/0 (socket says 172.21.15.32:56136) 2024-03-16T20:08:05.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5d3fff640 1 -- 172.21.15.32:0/3656392314 learned_addr learned my addr 172.21.15.32:0/3656392314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:05.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.680+0000 7fe5d8b50640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d415ac10 0x7fe5d41f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:05.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.684+0000 7fe5d3fff640 1 -- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d415ac10 msgr2=0x7fe5d41f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:05.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.684+0000 7fe5d3fff640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d415ac10 0x7fe5d41f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:05.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.684+0000 7fe5d3fff640 1 -- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d41540c0 msgr2=0x7fe5d40cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:05.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.684+0000 7fe5d3fff640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d41540c0 0x7fe5d40cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:05.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.684+0000 7fe5d3fff640 1 -- 172.21.15.32:0/3656392314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8009420 con 0x7fe5d40c8720 2024-03-16T20:08:05.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.688+0000 7fe5d3fff640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400e930 tx=0x7fe5c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4009690 con 0x7fe5d40c8720 2024-03-16T20:08:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c400ca00 con 0x7fe5d40c8720 2024-03-16T20:08:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400cd70 con 0x7fe5d40c8720 2024-03-16T20:08:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d4151bf0 con 0x7fe5d40c8720 2024-03-16T20:08:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d40ccdd0 con 0x7fe5d40c8720 2024-03-16T20:08:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:05.692+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d4070390 con 0x7fe5d40c8720 2024-03-16T20:08:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.016+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe5c4013470 con 0x7fe5d40c8720 2024-03-16T20:08:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.016+0000 7fe5d0ff9640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 0x7fe5c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.016+0000 7fe5d37fe640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 0x7fe5c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.016+0000 7fe5d37fe640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 0x7fe5c0069350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80ae000 tx=0x7fe5c80b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.040+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe5c40560b0 con 0x7fe5d40c8720 2024-03-16T20:08:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.040+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe5c4052350 con 0x7fe5d40c8720 2024-03-16T20:08:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.268+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5d4068000 con 0x7fe5d40c8720 2024-03-16T20:08:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5d0ff9640 1 -- 172.21.15.32:0/3656392314 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe5c4021090 con 0x7fe5d40c8720 2024-03-16T20:08:06.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:06.280 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 msgr2=0x7fe5c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 0x7fe5c0069350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80ae000 tx=0x7fe5c80b5040 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 msgr2=0x7fe5d414c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400e930 tx=0x7fe5c400edf0 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 shutdown_connections 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe5c0066e90 0x7fe5c0069350 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe5d415ac10 0x7fe5d41f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d41540c0 0x7fe5d40cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 --2- 172.21.15.32:0/3656392314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d40c8720 0x7fe5d414c500 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 >> 172.21.15.32:0/3656392314 conn(0x7fe5d4070f90 msgr2=0x7fe5d406fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 shutdown_connections 2024-03-16T20:08:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:06.292+0000 7fe5da5b3640 1 -- 172.21.15.32:0/3656392314 wait complete. 2024-03-16T20:08:06.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:06.902 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:07.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:08.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 Processor -- start 2024-03-16T20:08:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 -- start start 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16081540c0 0x7f160815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f160815ac10 0x7f16080c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160806c460 con 0x7f16080c8720 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160815a240 con 0x7f160815ac10 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f16101c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160806cf20 con 0x7f16081540c0 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f160d760640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f160815ac10 0x7f16080c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f160e762640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f16080c8720 0x7f16080c3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41592/0 (socket says 172.21.15.32:41592) 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f160e762640 1 -- 172.21.15.32:0/1630568952 learned_addr learned my addr 172.21.15.32:0/1630568952 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.024+0000 7f160df61640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16081540c0 0x7f160815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.028+0000 7f160d760640 1 -- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16081540c0 msgr2=0x7f160815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.028+0000 7f160d760640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16081540c0 0x7f160815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.028+0000 7f160d760640 1 -- 172.21.15.32:0/1630568952 >> v1:172.21.15.32:6789/0 conn(0x7f16080c8720 legacy=0x7f16080c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.028+0000 7f160d760640 1 -- 172.21.15.32:0/1630568952 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160806d200 con 0x7f160815ac10 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.032+0000 7f160d760640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f160815ac10 0x7f16080c81e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f15f80097c0 tx=0x7f15f801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f88ba9e750546480 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f160cf5f640 1 -- 172.21.15.32:0/1630568952 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80aa5f0 con 0x7f160815ac10 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f160cf5f640 1 -- 172.21.15.32:0/1630568952 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f80aa750 con 0x7f160815ac10 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f160cf5f640 1 -- 172.21.15.32:0/1630568952 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80aaae0 con 0x7f160815ac10 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f160815ac10 msgr2=0x7f16080c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f160815ac10 0x7f16080c81e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f15f80097c0 tx=0x7f15f801c470 comp rx=0 tx=0).stop 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- 172.21.15.32:0/1630568952 shutdown_connections 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f160815ac10 0x7f16080c81e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- 172.21.15.32:0/1630568952 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f16081540c0 0x7f160815a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- 172.21.15.32:0/1630568952 >> 172.21.15.32:0/1630568952 conn(0x7f1608070f90 msgr2=0x7f16080713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- 172.21.15.32:0/1630568952 shutdown_connections 2024-03-16T20:08:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- 172.21.15.32:0/1630568952 wait complete. 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 Processor -- start 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- start start 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16081540c0 0x7f16080cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f160815ac10 0x7f16081f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1608151b60 con 0x7f16080c8720 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1608151880 con 0x7f16081540c0 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.044+0000 7f16101c5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1608150380 con 0x7f160815ac10 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.048+0000 7f160df61640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.048+0000 7f160df61640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56144/0 (socket says 172.21.15.32:56144) 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.048+0000 7f160df61640 1 -- 172.21.15.32:0/3921234328 learned_addr learned my addr 172.21.15.32:0/3921234328 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.048+0000 7f160e762640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f160815ac10 0x7f16081f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.048+0000 7f160d760640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16081540c0 0x7f16080cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.052+0000 7f160df61640 1 -- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f160815ac10 msgr2=0x7f16081f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.052+0000 7f160df61640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f160815ac10 0x7f16081f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.052+0000 7f160df61640 1 -- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16081540c0 msgr2=0x7f16080cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.052+0000 7f160df61640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16081540c0 0x7f16080cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.052+0000 7f160df61640 1 -- 172.21.15.32:0/3921234328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009420 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.056+0000 7f160df61640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f160400e930 tx=0x7f160400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.060+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604009690 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.060+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160400ca00 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.060+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1608150660 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.060+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160400cd70 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.060+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16081f66a0 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.064+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1608070390 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.372+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1604013470 con 0x7f16080c8720 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.372+0000 7f15fe7fc640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 0x7f15ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.372+0000 7f160d760640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 0x7f15ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.376+0000 7f160d760640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 0x7f15ec069300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f15f80ae000 tx=0x7f15f80b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.400+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f16040560f0 con 0x7f16080c8720 2024-03-16T20:08:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.400+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1604052390 con 0x7f16080c8720 2024-03-16T20:08:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.596+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f160806ca90 con 0x7f16080c8720 2024-03-16T20:08:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f15fe7fc640 1 -- 172.21.15.32:0/3921234328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1604021020 con 0x7f16080c8720 2024-03-16T20:08:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:09.212 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 msgr2=0x7f15ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 0x7f15ec069300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f15f80ae000 tx=0x7f15f80b5040 comp rx=0 tx=0).stop 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 msgr2=0x7f160814e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f160400e930 tx=0x7f160400edf0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 shutdown_connections 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f15ec066e40 0x7f15ec069300 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f160815ac10 0x7f16081f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16081540c0 0x7f16080cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 --2- 172.21.15.32:0/3921234328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16080c8720 0x7f160814e8e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 >> 172.21.15.32:0/3921234328 conn(0x7f1608070f90 msgr2=0x7f160815d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 shutdown_connections 2024-03-16T20:08:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:08.612+0000 7f16101c5640 1 -- 172.21.15.32:0/3921234328 wait complete. 2024-03-16T20:08:09.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:09.820 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:10.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:10.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.944+0000 7f8741993640 1 Processor -- start 2024-03-16T20:08:10.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.944+0000 7f8741993640 1 -- start start 2024-03-16T20:08:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.948+0000 7f8741993640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f873c151060 0x7f873c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.948+0000 7f8741993640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c161550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.948+0000 7f8741993640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c070630 con 0x7f873c0c35b0 2024-03-16T20:08:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.948+0000 7f8741993640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c0c8ee0 con 0x7f873c151060 2024-03-16T20:08:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:10.948+0000 7f8741993640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0c8990 con 0x7f873c0cd210 2024-03-16T20:08:10.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.004+0000 7f873b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:10.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.008+0000 7f873affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f873c151060 0x7f873c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:10.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.008+0000 7f873b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c161550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:46784/0 (socket says 172.21.15.32:46784) 2024-03-16T20:08:10.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.008+0000 7f873b7fe640 1 -- 172.21.15.32:0/2819104404 learned_addr learned my addr 172.21.15.32:0/2819104404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:11.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.016+0000 7f873affd640 1 -- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 msgr2=0x7f873c161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:11.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.016+0000 7f873affd640 1 --2- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c161550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.016+0000 7f873affd640 1 -- 172.21.15.32:0/2819104404 >> v1:172.21.15.59:6789/0 conn(0x7f873c0c35b0 legacy=0x7f873c0c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.016+0000 7f873affd640 1 -- 172.21.15.32:0/2819104404 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0c82b0 con 0x7f873c151060 2024-03-16T20:08:11.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.020+0000 7f873affd640 1 --2- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c15a1c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f872c0097c0 tx=0x7f872c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0f8a019a1dac83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.024+0000 7f8739ffb640 1 -- 172.21.15.32:0/2819104404 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0a8990 con 0x7f873c151060 2024-03-16T20:08:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.024+0000 7f8739ffb640 1 -- 172.21.15.32:0/2819104404 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c0a8af0 con 0x7f873c151060 2024-03-16T20:08:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8739ffb640 1 -- 172.21.15.32:0/2819104404 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0a8e80 con 0x7f873c151060 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 msgr2=0x7f873c15a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c15a1c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f872c0097c0 tx=0x7f872c01ec90 comp rx=0 tx=0).stop 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- 172.21.15.32:0/2819104404 shutdown_connections 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- 172.21.15.32:0/2819104404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c15a1c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- 172.21.15.32:0/2819104404 >> 172.21.15.32:0/2819104404 conn(0x7f873c0cc5d0 msgr2=0x7f873c0cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- 172.21.15.32:0/2819104404 shutdown_connections 2024-03-16T20:08:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- 172.21.15.32:0/2819104404 wait complete. 2024-03-16T20:08:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 Processor -- start 2024-03-16T20:08:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- start start 2024-03-16T20:08:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c1eb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c14a900 0x7f873c145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c14bee0 con 0x7f873c14a900 2024-03-16T20:08:11.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c14b0e0 con 0x7f873c151060 2024-03-16T20:08:11.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.028+0000 7f8741993640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c14b380 con 0x7f873c0cd210 2024-03-16T20:08:11.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.032+0000 7f873affd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:11.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.032+0000 7f873affd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:46798/0 (socket says 172.21.15.32:46798) 2024-03-16T20:08:11.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.032+0000 7f873affd640 1 -- 172.21.15.32:0/2521572245 learned_addr learned my addr 172.21.15.32:0/2521572245 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:11.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.032+0000 7f873a7fc640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c1eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 -- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 msgr2=0x7f873c1eb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c1eb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 -- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c14a900 msgr2=0x7f873c145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c14a900 0x7f873c145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 -- 172.21.15.32:0/2521572245 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c009420 con 0x7f873c0cd210 2024-03-16T20:08:11.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.036+0000 7f873affd640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f872c01c8c0 tx=0x7f872c01ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.040+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0b0d90 con 0x7f873c0cd210 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.044+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c0b9430 con 0x7f873c0cd210 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.044+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c14b660 con 0x7f873c0cd210 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.044+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0b97e0 con 0x7f873c0cd210 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.044+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c1eee00 con 0x7f873c0cd210 2024-03-16T20:08:11.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.044+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c068000 con 0x7f873c0cd210 2024-03-16T20:08:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.308+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f872c0b7040 con 0x7f873c0cd210 2024-03-16T20:08:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.308+0000 7f8723fff640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 0x7f8728069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.308+0000 7f873a7fc640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 0x7f8728069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:11.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.312+0000 7f873a7fc640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 0x7f8728069350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f873c0c3430 tx=0x7f8730009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.316+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f872c0f4160 con 0x7f873c0cd210 2024-03-16T20:08:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.340+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f872c0efd00 con 0x7f873c0cd210 2024-03-16T20:08:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.568+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f873c070b50 con 0x7f873c0cd210 2024-03-16T20:08:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.600+0000 7f8723fff640 1 -- 172.21.15.32:0/2521572245 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f872c0b5090 con 0x7f873c0cd210 2024-03-16T20:08:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:11.591 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:11.591 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 msgr2=0x7f8728069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 0x7f8728069350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f873c0c3430 tx=0x7f8730009040 comp rx=0 tx=0).stop 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 msgr2=0x7f873c1ef8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f872c01c8c0 tx=0x7f872c01ba30 comp rx=0 tx=0).stop 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 shutdown_connections 2024-03-16T20:08:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8728066e90 0x7f8728069350 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c14a900 0x7f873c145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873c151060 0x7f873c1eb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 --2- 172.21.15.32:0/2521572245 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f873c0cd210 0x7f873c1ef8e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 >> 172.21.15.32:0/2521572245 conn(0x7f873c0cc5d0 msgr2=0x7f873c150280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 shutdown_connections 2024-03-16T20:08:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:11.604+0000 7f8741993640 1 -- 172.21.15.32:0/2521572245 wait complete. 2024-03-16T20:08:12.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:12.215 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:12.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:08:12.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:08:13.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:13.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 Processor -- start 2024-03-16T20:08:13.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 -- start start 2024-03-16T20:08:13.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c36b0 0x7f396c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f396c0c8980 0x7f396c15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0c8ec0 con 0x7f396c151000 2024-03-16T20:08:13.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c072f10 con 0x7f396c0c8980 2024-03-16T20:08:13.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.396+0000 7f3970825640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0c5000 con 0x7f396c0c36b0 2024-03-16T20:08:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.400+0000 7f396ad9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f396c0c8980 0x7f396c15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.400+0000 7f396ad9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f396c0c8980 0x7f396c15a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57900/0 (socket says 172.21.15.32:57900) 2024-03-16T20:08:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.400+0000 7f396ad9d640 1 -- 172.21.15.32:0/3078505985 learned_addr learned my addr 172.21.15.32:0/3078505985 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.404+0000 7f396ad9d640 1 -- 172.21.15.32:0/3078505985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c36b0 msgr2=0x7f396c0c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.404+0000 7f396ad9d640 1 --2- 172.21.15.32:0/3078505985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c36b0 0x7f396c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.404+0000 7f396ad9d640 1 -- 172.21.15.32:0/3078505985 >> v1:172.21.15.32:6789/0 conn(0x7f396c151000 legacy=0x7f396c0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.404+0000 7f396ad9d640 1 -- 172.21.15.32:0/3078505985 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c072950 con 0x7f396c0c8980 2024-03-16T20:08:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.408+0000 7f396ad9d640 1 --2- 172.21.15.32:0/3078505985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c0c8980 0x7f396c15a240 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f396401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41405cfccc6eaa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f396959a640 1 -- 172.21.15.32:0/3078505985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640a5900 con 0x7f396c0c8980 2024-03-16T20:08:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f396959a640 1 -- 172.21.15.32:0/3078505985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39640a5a60 con 0x7f396c0c8980 2024-03-16T20:08:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f396959a640 1 -- 172.21.15.32:0/3078505985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640a5df0 con 0x7f396c0c8980 2024-03-16T20:08:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 -- 172.21.15.32:0/3078505985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c0c8980 msgr2=0x7f396c15a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 --2- 172.21.15.32:0/3078505985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c0c8980 0x7f396c15a240 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f396401ca30 comp rx=0 tx=0).stop 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 -- 172.21.15.32:0/3078505985 shutdown_connections 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 --2- 172.21.15.32:0/3078505985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c0c8980 0x7f396c15a240 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 --2- 172.21.15.32:0/3078505985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c36b0 0x7f396c0c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 -- 172.21.15.32:0/3078505985 >> 172.21.15.32:0/3078505985 conn(0x7f396c06c0d0 msgr2=0x7f396c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 -- 172.21.15.32:0/3078505985 shutdown_connections 2024-03-16T20:08:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.412+0000 7f3970825640 1 -- 172.21.15.32:0/3078505985 wait complete. 2024-03-16T20:08:13.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 Processor -- start 2024-03-16T20:08:13.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 -- start start 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0c36b0 0x7f396c147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c8980 0x7f396c1484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c144ee0 con 0x7f396c151000 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1454a0 con 0x7f396c0c36b0 2024-03-16T20:08:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f3970825640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c158fc0 con 0x7f396c0c8980 2024-03-16T20:08:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f396ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f396ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57904/0 (socket says 172.21.15.32:57904) 2024-03-16T20:08:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f396ad9d640 1 -- 172.21.15.32:0/2976467904 learned_addr learned my addr 172.21.15.32:0/2976467904 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.416+0000 7f396a59c640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0c36b0 0x7f396c147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:13.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 -- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c8980 msgr2=0x7f396c1484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c8980 0x7f396c1484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 -- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0c36b0 msgr2=0x7f396c147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0c36b0 0x7f396c147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 -- 172.21.15.32:0/2976467904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009b30 con 0x7f396c151000 2024-03-16T20:08:13.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.424+0000 7f396ad9d640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f39640197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640b8880 con 0x7f396c151000 2024-03-16T20:08:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39640b89e0 con 0x7f396c151000 2024-03-16T20:08:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640b8d90 con 0x7f396c151000 2024-03-16T20:08:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c1592a0 con 0x7f396c151000 2024-03-16T20:08:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c145130 con 0x7f396c151000 2024-03-16T20:08:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.432+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c068000 con 0x7f396c151000 2024-03-16T20:08:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.456+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f39640ca430 con 0x7f396c151000 2024-03-16T20:08:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.456+0000 7f395affd640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 0x7f394c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.456+0000 7f396a59c640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 0x7f394c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:13.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.456+0000 7f396a59c640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 0x7f394c069350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f395c001770 tx=0x7f395c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:13.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.728+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3964009250 con 0x7f396c151000 2024-03-16T20:08:13.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.728+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f39640ed300 con 0x7f396c151000 2024-03-16T20:08:13.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.968+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f396c1455e0 con 0x7f396c151000 2024-03-16T20:08:13.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.992+0000 7f395affd640 1 -- 172.21.15.32:0/2976467904 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f39640bc020 con 0x7f396c151000 2024-03-16T20:08:13.979 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:13.979 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:13.980 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 msgr2=0x7f394c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 0x7f394c069350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f395c001770 tx=0x7f395c009040 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 msgr2=0x7f396c145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f39640197d0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 shutdown_connections 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f394c066e90 0x7f394c069350 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f396c151000 0x7f396c145930 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f396c0c8980 0x7f396c1484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 --2- 172.21.15.32:0/2976467904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0c36b0 0x7f396c147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 >> 172.21.15.32:0/2976467904 conn(0x7f396c06c0d0 msgr2=0x7f396c15ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 shutdown_connections 2024-03-16T20:08:13.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:13.996+0000 7f3970825640 1 -- 172.21.15.32:0/2976467904 wait complete. 2024-03-16T20:08:14.603 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:14.603 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:15.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.724+0000 7f8caa816640 1 Processor -- start 2024-03-16T20:08:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.724+0000 7f8caa816640 1 -- start start 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8caa816640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 0x7f8ca4165780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8caa816640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca4070c90 con 0x7f8ca4153fa0 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8caa816640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40c7900 con 0x7f8ca4070840 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8caa816640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40c3f00 con 0x7f8ca40c7470 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8ca3fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ca4153fa0 0x7f8ca415d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54874/0 (socket says 172.21.15.32:54874) 2024-03-16T20:08:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8ca3fff640 1 -- 172.21.15.32:0/3740637050 learned_addr learned my addr 172.21.15.32:0/3740637050 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.728+0000 7f8ca8db3640 1 --2- 172.21.15.32:0/3740637050 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 0x7f8ca4165780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:15.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.732+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60444543 0 0) 0x7f8c94001120 con 0x7f8ca4153fa0 2024-03-16T20:08:15.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.732+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c003530 con 0x7f8ca4153fa0 2024-03-16T20:08:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.732+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089211173 0 0) 0x7f8c98001120 con 0x7f8ca4070840 2024-03-16T20:08:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.732+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c94001120 con 0x7f8ca4070840 2024-03-16T20:08:15.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.736+0000 7f8ca8db3640 1 -- 172.21.15.32:0/3740637050 >> v1:172.21.15.59:6789/0 conn(0x7f8ca4070840 legacy=0x7f8ca40724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:15.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.736+0000 7f8ca8db3640 1 -- 172.21.15.32:0/3740637050 >> v1:172.21.15.32:6789/0 conn(0x7f8ca4153fa0 legacy=0x7f8ca415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:15.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.736+0000 7f8ca8db3640 1 -- 172.21.15.32:0/3740637050 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40c8590 con 0x7f8ca40c7470 2024-03-16T20:08:15.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.740+0000 7f8ca8db3640 1 --2- 172.21.15.32:0/3740637050 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 0x7f8ca4165780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009dc0 tx=0x7f8c9c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378cdab0a2a08672 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c0af600 con 0x7f8ca40c7470 2024-03-16T20:08:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c0ab050 con 0x7f8ca40c7470 2024-03-16T20:08:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8ca2ffd640 1 -- 172.21.15.32:0/3740637050 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c0af980 con 0x7f8ca40c7470 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 -- 172.21.15.32:0/3740637050 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 msgr2=0x7f8ca4165780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 --2- 172.21.15.32:0/3740637050 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 0x7f8ca4165780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009dc0 tx=0x7f8c9c01f170 comp rx=0 tx=0).stop 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 -- 172.21.15.32:0/3740637050 shutdown_connections 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 --2- 172.21.15.32:0/3740637050 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca40c7470 0x7f8ca4165780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 -- 172.21.15.32:0/3740637050 >> 172.21.15.32:0/3740637050 conn(0x7f8ca406c0d0 msgr2=0x7f8ca406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 -- 172.21.15.32:0/3740637050 shutdown_connections 2024-03-16T20:08:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 -- 172.21.15.32:0/3740637050 wait complete. 2024-03-16T20:08:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.748+0000 7f8caa816640 1 Processor -- start 2024-03-16T20:08:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 -- start start 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ca40c7470 0x7f8ca4148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca4153fa0 0x7f8ca4148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4152dd0 con 0x7f8ca4070840 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4152a10 con 0x7f8ca40c7470 2024-03-16T20:08:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8caa816640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca415ded0 con 0x7f8ca4153fa0 2024-03-16T20:08:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8ca3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8ca8db3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca4153fa0 0x7f8ca4148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8ca37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ca40c7470 0x7f8ca4148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8ca3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57912/0 (socket says 172.21.15.32:57912) 2024-03-16T20:08:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.752+0000 7f8ca3fff640 1 -- 172.21.15.32:0/4073560570 learned_addr learned my addr 172.21.15.32:0/4073560570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:15.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.756+0000 7f8ca3fff640 1 -- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca4153fa0 msgr2=0x7f8ca4148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:15.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.756+0000 7f8ca3fff640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca4153fa0 0x7f8ca4148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.760+0000 7f8ca3fff640 1 -- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ca40c7470 msgr2=0x7f8ca4148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.760+0000 7f8ca3fff640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ca40c7470 0x7f8ca4148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.760+0000 7f8ca3fff640 1 -- 172.21.15.32:0/4073560570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c009a70 con 0x7f8ca4070840 2024-03-16T20:08:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.760+0000 7f8ca3fff640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400c760 tx=0x7f8c9400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:15.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.764+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94005cf0 con 0x7f8ca4070840 2024-03-16T20:08:15.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.768+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c94005e50 con 0x7f8ca4070840 2024-03-16T20:08:15.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.768+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94007290 con 0x7f8ca4070840 2024-03-16T20:08:15.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.768+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca415c120 con 0x7f8ca4070840 2024-03-16T20:08:15.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.768+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca415c6e0 con 0x7f8ca4070840 2024-03-16T20:08:15.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.768+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca4068000 con 0x7f8ca4070840 2024-03-16T20:08:15.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.804+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c940073f0 con 0x7f8ca4070840 2024-03-16T20:08:15.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.804+0000 7f8ca0ff9640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 0x7f8c90069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:15.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.808+0000 7f8ca37fe640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 0x7f8c90069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:15.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.808+0000 7f8ca37fe640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 0x7f8c90069330 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40c6660 tx=0x7f8c98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:15.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:15.812+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8c94054f00 con 0x7f8ca4070840 2024-03-16T20:08:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.104+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8c940511a0 con 0x7f8ca4070840 2024-03-16T20:08:16.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.304+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ca4070c90 con 0x7f8ca4070840 2024-03-16T20:08:16.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8ca0ff9640 1 -- 172.21.15.32:0/4073560570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8ca4070c90 con 0x7f8ca4070840 2024-03-16T20:08:16.315 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:16.315 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 msgr2=0x7f8c90069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 0x7f8c90069330 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40c6660 tx=0x7f8c98009040 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 msgr2=0x7f8ca415cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400c760 tx=0x7f8c9400cc20 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 shutdown_connections 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8c90066e70 0x7f8c90069330 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ca4153fa0 0x7f8ca4148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ca40c7470 0x7f8ca4148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 --2- 172.21.15.32:0/4073560570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4070840 0x7f8ca415cb70 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.328+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 >> 172.21.15.32:0/4073560570 conn(0x7f8ca406c0d0 msgr2=0x7f8ca4149a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.332+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 shutdown_connections 2024-03-16T20:08:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:16.332+0000 7f8caa816640 1 -- 172.21.15.32:0/4073560570 wait complete. 2024-03-16T20:08:16.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:16.945 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:17.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:18.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.064+0000 7ff0ff6ef640 1 Processor -- start 2024-03-16T20:08:18.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.064+0000 7ff0ff6ef640 1 -- start start 2024-03-16T20:08:18.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0ff6ef640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0f80c6040 0x7ff0f80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0ff6ef640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 0x7ff0f8157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0ff6ef640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f8070760 con 0x7ff0f806ba30 2024-03-16T20:08:18.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0ff6ef640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f80731c0 con 0x7ff0f80c6040 2024-03-16T20:08:18.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0ff6ef640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f80734a0 con 0x7ff0f80c6420 2024-03-16T20:08:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0fcc8a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 0x7ff0f8157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0fdc8c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0f806ba30 0x7ff0f81516d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54878/0 (socket says 172.21.15.32:54878) 2024-03-16T20:08:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0fdc8c640 1 -- 172.21.15.32:0/3498646233 learned_addr learned my addr 172.21.15.32:0/3498646233 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.068+0000 7ff0fd48b640 1 --2- 172.21.15.32:0/3498646233 >> v2:172.21.15.59:3300/0 conn(0x7ff0f80c6040 0x7ff0f80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0effff640 1 -- 172.21.15.32:0/3498646233 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000803219 0 0) 0x7ff0f8070760 con 0x7ff0f806ba30 2024-03-16T20:08:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0effff640 1 -- 172.21.15.32:0/3498646233 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e4003530 con 0x7ff0f806ba30 2024-03-16T20:08:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0fcc8a640 1 -- 172.21.15.32:0/3498646233 >> v2:172.21.15.59:3300/0 conn(0x7ff0f80c6040 msgr2=0x7ff0f80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0fcc8a640 1 --2- 172.21.15.32:0/3498646233 >> v2:172.21.15.59:3300/0 conn(0x7ff0f80c6040 0x7ff0f80c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0fcc8a640 1 -- 172.21.15.32:0/3498646233 >> v1:172.21.15.32:6789/0 conn(0x7ff0f806ba30 legacy=0x7ff0f81516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.072+0000 7ff0fcc8a640 1 -- 172.21.15.32:0/3498646233 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f806fe30 con 0x7ff0f80c6420 2024-03-16T20:08:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.076+0000 7ff0fcc8a640 1 --2- 172.21.15.32:0/3498646233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 0x7ff0f8157c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097c0 tx=0x7ff0e801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d043139886913b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0effff640 1 -- 172.21.15.32:0/3498646233 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80ad680 con 0x7ff0f80c6420 2024-03-16T20:08:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0effff640 1 -- 172.21.15.32:0/3498646233 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80ad7e0 con 0x7ff0f80c6420 2024-03-16T20:08:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0effff640 1 -- 172.21.15.32:0/3498646233 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80adb70 con 0x7ff0f80c6420 2024-03-16T20:08:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/3498646233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 msgr2=0x7ff0f8157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/3498646233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 0x7ff0f8157c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097c0 tx=0x7ff0e801c490 comp rx=0 tx=0).stop 2024-03-16T20:08:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/3498646233 shutdown_connections 2024-03-16T20:08:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/3498646233 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6420 0x7ff0f8157c80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/3498646233 >> v2:172.21.15.59:3300/0 conn(0x7ff0f80c6040 0x7ff0f80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/3498646233 >> 172.21.15.32:0/3498646233 conn(0x7ff0f806c0d0 msgr2=0x7ff0f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/3498646233 shutdown_connections 2024-03-16T20:08:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/3498646233 wait complete. 2024-03-16T20:08:18.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 Processor -- start 2024-03-16T20:08:18.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.084+0000 7ff0ff6ef640 1 -- start start 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0f806ba30 0x7ff0f81410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6040 0x7ff0f80caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f81f18d0 con 0x7ff0f80c6420 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f81f1d60 con 0x7ff0f806ba30 2024-03-16T20:08:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0ff6ef640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8141c10 con 0x7ff0f80c6040 2024-03-16T20:08:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0fdc8c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0fcc8a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6040 0x7ff0f80caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0fdc8c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57922/0 (socket says 172.21.15.32:57922) 2024-03-16T20:08:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0fdc8c640 1 -- 172.21.15.32:0/188037350 learned_addr learned my addr 172.21.15.32:0/188037350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.088+0000 7ff0fd48b640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0f806ba30 0x7ff0f81410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.092+0000 7ff0fdc8c640 1 -- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6040 msgr2=0x7ff0f80caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.092+0000 7ff0fdc8c640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6040 0x7ff0f80caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.092+0000 7ff0fdc8c640 1 -- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0f806ba30 msgr2=0x7ff0f81410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.092+0000 7ff0fdc8c640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0f806ba30 0x7ff0f81410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.092+0000 7ff0fdc8c640 1 -- 172.21.15.32:0/188037350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8009420 con 0x7ff0f80c6420 2024-03-16T20:08:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.096+0000 7ff0fdc8c640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000cb10 tx=0x7ff0f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.100+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000ac70 con 0x7ff0f80c6420 2024-03-16T20:08:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.100+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f000add0 con 0x7ff0f80c6420 2024-03-16T20:08:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.100+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8141930 con 0x7ff0f80c6420 2024-03-16T20:08:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.100+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0012620 con 0x7ff0f80c6420 2024-03-16T20:08:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.100+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f81f1280 con 0x7ff0f80c6420 2024-03-16T20:08:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.104+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8151480 con 0x7ff0f80c6420 2024-03-16T20:08:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.120+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0f0007070 con 0x7ff0f80c6420 2024-03-16T20:08:18.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.124+0000 7ff0edffb640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 0x7ff0dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:18.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.124+0000 7ff0fd48b640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 0x7ff0dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:18.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.124+0000 7ff0fd48b640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 0x7ff0dc069350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009b60 tx=0x7ff0f4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.400+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff0f0055a30 con 0x7ff0f80c6420 2024-03-16T20:08:18.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.400+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff0f0051cd0 con 0x7ff0f80c6420 2024-03-16T20:08:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.636+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0f806fa50 con 0x7ff0f80c6420 2024-03-16T20:08:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.664+0000 7ff0edffb640 1 -- 172.21.15.32:0/188037350 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7ff0f0017070 con 0x7ff0f80c6420 2024-03-16T20:08:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:18.652 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 msgr2=0x7ff0dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 0x7ff0dc069350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009b60 tx=0x7ff0f4009190 comp rx=0 tx=0).stop 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 msgr2=0x7ff0f80cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000cb10 tx=0x7ff0f000a060 comp rx=0 tx=0).stop 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 shutdown_connections 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff0dc066e90 0x7ff0dc069350 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0f80c6420 0x7ff0f80cbf70 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff0f80c6040 0x7ff0f80caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 --2- 172.21.15.32:0/188037350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0f806ba30 0x7ff0f81410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 >> 172.21.15.32:0/188037350 conn(0x7ff0f806c0d0 msgr2=0x7ff0f8147ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 shutdown_connections 2024-03-16T20:08:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:18.668+0000 7ff0ff6ef640 1 -- 172.21.15.32:0/188037350 wait complete. 2024-03-16T20:08:19.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:19.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:19.894 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:13:03.976 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:19.966 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:13:04.048 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:19.974 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:13:03.994 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:19.982 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:13:04.001 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.045 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:13:04.065 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.047 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:13:04.067 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.092 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:13:04.174 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.121 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:13:04.141 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.166 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:13:04.185 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:08:20.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:20.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.460+0000 7f4ff5e67640 1 Processor -- start 2024-03-16T20:08:20.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.460+0000 7f4ff5e67640 1 -- start start 2024-03-16T20:08:20.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4ff5e67640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4ff00ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4ff5e67640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00cbfa0 con 0x7f4ff00cc590 2024-03-16T20:08:20.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4ff5e67640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff015a140 con 0x7f4ff00caab0 2024-03-16T20:08:20.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4ff5e67640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff015e2c0 con 0x7f4ff00cc970 2024-03-16T20:08:20.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4fef7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ff00cc590 0x7f4ff015dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54520/0 (socket says 172.21.15.32:54520) 2024-03-16T20:08:20.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.464+0000 7f4fef7fe640 1 -- 172.21.15.32:0/3851576733 learned_addr learned my addr 172.21.15.32:0/3851576733 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:20.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402307737 0 0) 0x7f4ff00cbfa0 con 0x7f4ff00cc590 2024-03-16T20:08:20.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8003530 con 0x7f4ff00cc590 2024-03-16T20:08:20.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4feeffd640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4ff00ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611875541 0 0) 0x7f4fd8003530 con 0x7f4ff00cc590 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 msgr2=0x7f4ff00ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4ff00ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 >> v1:172.21.15.59:6789/0 conn(0x7f4ff00caab0 legacy=0x7f4ff0164740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff015a3e0 con 0x7f4ff00cc590 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f4fe0001680 con 0x7f4ff00cc590 2024-03-16T20:08:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 >> v1:172.21.15.32:6789/0 conn(0x7f4ff00cc590 legacy=0x7f4ff015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80039b0 0x7f4fd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd80047d0 0x7f4fd8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4fd800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff015e2c0 con 0x7f4fd80047d0 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd8005580 con 0x7f4fd80039b0 2024-03-16T20:08:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.468+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd800bed0 con 0x7f4ff00cc970 2024-03-16T20:08:20.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.472+0000 7f4feeffd640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd80047d0 0x7f4fd8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feffff640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4fd800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feeffd640 1 -- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 msgr2=0x7f4fd800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feeffd640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff00cc970 0x7f4fd800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feeffd640 1 -- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80039b0 msgr2=0x7f4fd8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feeffd640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80039b0 0x7f4fd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.476+0000 7f4feeffd640 1 -- 172.21.15.32:0/3851576733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff015a3e0 con 0x7f4fd80047d0 2024-03-16T20:08:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.480+0000 7f4feeffd640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd80047d0 0x7f4fd8004c50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400a9c0 tx=0x7f4fe401c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45bcfe7bdf0e952 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40aa620 con 0x7f4fd80047d0 2024-03-16T20:08:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe40aa780 con 0x7f4fd80047d0 2024-03-16T20:08:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4fee7fc640 1 -- 172.21.15.32:0/3851576733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40aab70 con 0x7f4fd80047d0 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- 172.21.15.32:0/3851576733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd80047d0 msgr2=0x7f4fd8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/3851576733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd80047d0 0x7f4fd8004c50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400a9c0 tx=0x7f4fe401c570 comp rx=0 tx=0).stop 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4feffff640 1 -- 172.21.15.32:0/3851576733 reap_dead start 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- 172.21.15.32:0/3851576733 shutdown_connections 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- 172.21.15.32:0/3851576733 >> 172.21.15.32:0/3851576733 conn(0x7f4ff0066ea0 msgr2=0x7f4ff00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- 172.21.15.32:0/3851576733 shutdown_connections 2024-03-16T20:08:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- 172.21.15.32:0/3851576733 wait complete. 2024-03-16T20:08:20.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 Processor -- start 2024-03-16T20:08:20.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- start start 2024-03-16T20:08:20.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80047d0 0x7f4ff014a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff0159990 0x7f4ff014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff014b590 con 0x7f4ff0151340 2024-03-16T20:08:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00b2cd0 con 0x7f4fd80047d0 2024-03-16T20:08:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.488+0000 7f4ff5e67640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00b2fb0 con 0x7f4ff0159990 2024-03-16T20:08:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.492+0000 7f4feffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.492+0000 7f4feffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36754/0 (socket says 172.21.15.32:36754) 2024-03-16T20:08:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.492+0000 7f4feffff640 1 -- 172.21.15.32:0/445975040 learned_addr learned my addr 172.21.15.32:0/445975040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.492+0000 7f4fef7fe640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80047d0 0x7f4ff014a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.496+0000 7f4feffff640 1 -- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff0159990 msgr2=0x7f4ff014a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.496+0000 7f4feffff640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff0159990 0x7f4ff014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.496+0000 7f4feffff640 1 -- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80047d0 msgr2=0x7f4ff014a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.496+0000 7f4feffff640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80047d0 0x7f4ff014a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:20.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.496+0000 7f4feffff640 1 -- 172.21.15.32:0/445975040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00b3290 con 0x7f4ff0151340 2024-03-16T20:08:20.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.500+0000 7f4feffff640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00cd160 tx=0x7f4fe800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:20.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.504+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8003b80 con 0x7f4ff0151340 2024-03-16T20:08:20.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.504+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8003d00 con 0x7f4ff0151340 2024-03-16T20:08:20.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.504+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80052b0 con 0x7f4ff0151340 2024-03-16T20:08:20.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.508+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff00b3570 con 0x7f4ff0151340 2024-03-16T20:08:20.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.508+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff00b3b30 con 0x7f4ff0151340 2024-03-16T20:08:20.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.508+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff00b3cc0 con 0x7f4ff0151340 2024-03-16T20:08:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.804+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4fe8005410 con 0x7f4ff0151340 2024-03-16T20:08:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.804+0000 7f4fd7fff640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 0x7f4fdc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.804+0000 7f4fef7fe640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 0x7f4fdc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.804+0000 7f4fef7fe640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 0x7f4fdc069300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0001ec0 tx=0x7f4fe00016d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:20.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.824+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4fe8055b40 con 0x7f4ff0151340 2024-03-16T20:08:20.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:20.824+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4fe80516e0 con 0x7f4ff0151340 2024-03-16T20:08:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.052+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff01f0980 con 0x7f4ff0151340 2024-03-16T20:08:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.076+0000 7f4fd7fff640 1 -- 172.21.15.32:0/445975040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4fe8018020 con 0x7f4ff0151340 2024-03-16T20:08:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:21.065 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 msgr2=0x7f4fdc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 0x7f4fdc069300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0001ec0 tx=0x7f4fe00016d0 comp rx=0 tx=0).stop 2024-03-16T20:08:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 msgr2=0x7f4ff01f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00cd160 tx=0x7f4fe800dc00 comp rx=0 tx=0).stop 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 shutdown_connections 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fdc066e40 0x7f4fdc069300 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ff0151340 0x7f4ff01f0bc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4ff0159990 0x7f4ff014a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 --2- 172.21.15.32:0/445975040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd80047d0 0x7f4ff014a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 >> 172.21.15.32:0/445975040 conn(0x7f4ff0066ea0 msgr2=0x7f4ff015b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 shutdown_connections 2024-03-16T20:08:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:21.080+0000 7f4ff5e67640 1 -- 172.21.15.32:0/445975040 wait complete. 2024-03-16T20:08:21.687 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:21.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:22.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 Processor -- start 2024-03-16T20:08:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 -- start start 2024-03-16T20:08:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1dc0157180 0x7f1dc006be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc006bca0 con 0x7f1dc0150710 2024-03-16T20:08:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc0156cf0 con 0x7f1dc0157180 2024-03-16T20:08:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.808+0000 7f1dc7488640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc006c500 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.812+0000 7f1dc5a25640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.812+0000 7f1dc4a23640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1dc0157180 0x7f1dc006be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.812+0000 7f1dc5a25640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:47248/0 (socket says 172.21.15.32:47248) 2024-03-16T20:08:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.812+0000 7f1dc5a25640 1 -- 172.21.15.32:0/4281723843 learned_addr learned my addr 172.21.15.32:0/4281723843 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.816+0000 7f1dc5a25640 1 -- 172.21.15.32:0/4281723843 >> v2:172.21.15.59:3300/0 conn(0x7f1dc0157180 msgr2=0x7f1dc006be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.816+0000 7f1dc5a25640 1 --2- 172.21.15.32:0/4281723843 >> v2:172.21.15.59:3300/0 conn(0x7f1dc0157180 0x7f1dc006be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.816+0000 7f1dc5a25640 1 -- 172.21.15.32:0/4281723843 >> v1:172.21.15.32:6789/0 conn(0x7f1dc0150710 legacy=0x7f1dc006d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.816+0000 7f1dc5a25640 1 -- 172.21.15.32:0/4281723843 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc006c7a0 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.824+0000 7f1dc5a25640 1 --2- 172.21.15.32:0/4281723843 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1db800d150 tx=0x7f1db801ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=866fdcc41a507219 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:22.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.832+0000 7f1db7fff640 1 -- 172.21.15.32:0/4281723843 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80a7bf0 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.832+0000 7f1db7fff640 1 -- 172.21.15.32:0/4281723843 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80ab070 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.832+0000 7f1db7fff640 1 -- 172.21.15.32:0/4281723843 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b1400 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- 172.21.15.32:0/4281723843 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 msgr2=0x7f1dc0072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- 172.21.15.32:0/4281723843 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1db800d150 tx=0x7f1db801ec30 comp rx=0 tx=0).stop 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- 172.21.15.32:0/4281723843 shutdown_connections 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- 172.21.15.32:0/4281723843 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0072a10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- 172.21.15.32:0/4281723843 >> v2:172.21.15.59:3300/0 conn(0x7f1dc0157180 0x7f1dc006be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- 172.21.15.32:0/4281723843 >> 172.21.15.32:0/4281723843 conn(0x7f1dc006afc0 msgr2=0x7f1dc006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- 172.21.15.32:0/4281723843 shutdown_connections 2024-03-16T20:08:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- 172.21.15.32:0/4281723843 wait complete. 2024-03-16T20:08:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 Processor -- start 2024-03-16T20:08:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- start start 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0157180 0x7f1dc0145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc0143c10 con 0x7f1dc0150710 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc01f07f0 con 0x7f1dc0157180 2024-03-16T20:08:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc7488640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc01415e0 con 0x7f1dc00b1ea0 2024-03-16T20:08:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc4a23640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc4a23640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36756/0 (socket says 172.21.15.32:36756) 2024-03-16T20:08:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc4a23640 1 -- 172.21.15.32:0/3593502296 learned_addr learned my addr 172.21.15.32:0/3593502296 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.836+0000 7f1dc5a25640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0157180 0x7f1dc0145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.840+0000 7f1dc5224640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.844+0000 7f1dc4a23640 1 -- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 msgr2=0x7f1dc0143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:22.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.844+0000 7f1dc4a23640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:22.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.844+0000 7f1dc4a23640 1 -- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0157180 msgr2=0x7f1dc0145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.844+0000 7f1dc4a23640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0157180 0x7f1dc0145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.844+0000 7f1dc4a23640 1 -- 172.21.15.32:0/3593502296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db800bd40 con 0x7f1dc0150710 2024-03-16T20:08:22.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.848+0000 7f1dc4a23640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e930 tx=0x7f1db000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0009690 con 0x7f1dc0150710 2024-03-16T20:08:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db000ca00 con 0x7f1dc0150710 2024-03-16T20:08:22.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000cd70 con 0x7f1dc0150710 2024-03-16T20:08:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc01444f0 con 0x7f1dc0150710 2024-03-16T20:08:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc0141b60 con 0x7f1dc0150710 2024-03-16T20:08:22.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.852+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc0068000 con 0x7f1dc0150710 2024-03-16T20:08:22.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.888+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1db0013470 con 0x7f1dc0150710 2024-03-16T20:08:22.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.888+0000 7f1db5ffb640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 0x7f1da4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:22.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.888+0000 7f1dc5224640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 0x7f1da4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:22.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.892+0000 7f1dc5224640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 0x7f1da4069330 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc001810 tx=0x7f1dbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:22.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:22.892+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1db00560f0 con 0x7f1dc0150710 2024-03-16T20:08:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.184+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1db0052390 con 0x7f1dc0150710 2024-03-16T20:08:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.396+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dc006eca0 con 0x7f1dc0150710 2024-03-16T20:08:23.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.468+0000 7f1db5ffb640 1 -- 172.21.15.32:0/3593502296 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1db0021090 con 0x7f1dc0150710 2024-03-16T20:08:23.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:23.459 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:23.459 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 msgr2=0x7f1da4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 0x7f1da4069330 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc001810 tx=0x7f1dbc009040 comp rx=0 tx=0).stop 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 msgr2=0x7f1dc01f02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e930 tx=0x7f1db000edf0 comp rx=0 tx=0).stop 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 shutdown_connections 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1da4066e70 0x7f1da4069330 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:23.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0157180 0x7f1dc0145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:23.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dc0150710 0x7f1dc01f02b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:23.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 --2- 172.21.15.32:0/3593502296 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1dc00b1ea0 0x7f1dc0143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:23.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 >> 172.21.15.32:0/3593502296 conn(0x7f1dc006afc0 msgr2=0x7f1dc014eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:23.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 shutdown_connections 2024-03-16T20:08:23.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:23.472+0000 7f1dc7488640 1 -- 172.21.15.32:0/3593502296 wait complete. 2024-03-16T20:08:24.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:24.087 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:25.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:25.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.204+0000 7f204f00a640 1 Processor -- start 2024-03-16T20:08:25.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.204+0000 7f204f00a640 1 -- start start 2024-03-16T20:08:25.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f204f00a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:25.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f204f00a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2048158c80 con 0x7f20480c97b0 2024-03-16T20:08:25.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f204f00a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204815bfc0 con 0x7f204815b2a0 2024-03-16T20:08:25.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f204f00a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20480b0ad0 con 0x7f2048151ef0 2024-03-16T20:08:25.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f204cda6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:25.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f2047fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20480c97b0 0x7f20480c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54532/0 (socket says 172.21.15.32:54532) 2024-03-16T20:08:25.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.208+0000 7f2047fff640 1 -- 172.21.15.32:0/475373212 learned_addr learned my addr 172.21.15.32:0/475373212 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:25.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.212+0000 7f20477fe640 1 -- 172.21.15.32:0/475373212 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638100864 0 0) 0x7f2048158c80 con 0x7f20480c97b0 2024-03-16T20:08:25.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.212+0000 7f20477fe640 1 -- 172.21.15.32:0/475373212 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2038003530 con 0x7f20480c97b0 2024-03-16T20:08:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.212+0000 7f204cda6640 1 -- 172.21.15.32:0/475373212 >> v1:172.21.15.59:6789/0 conn(0x7f204815b2a0 legacy=0x7f204815b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.212+0000 7f204cda6640 1 -- 172.21.15.32:0/475373212 >> v1:172.21.15.32:6789/0 conn(0x7f20480c97b0 legacy=0x7f20480c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.212+0000 7f204cda6640 1 -- 172.21.15.32:0/475373212 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2048157ec0 con 0x7f2048151ef0 2024-03-16T20:08:25.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.216+0000 7f204cda6640 1 --2- 172.21.15.32:0/475373212 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480c9270 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f203c0097f0 tx=0x7f203c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae3dd48c81a61608 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f20477fe640 1 -- 172.21.15.32:0/475373212 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0a89d0 con 0x7f2048151ef0 2024-03-16T20:08:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f20477fe640 1 -- 172.21.15.32:0/475373212 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c0a8b30 con 0x7f2048151ef0 2024-03-16T20:08:25.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f20477fe640 1 -- 172.21.15.32:0/475373212 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0b0440 con 0x7f2048151ef0 2024-03-16T20:08:25.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 -- 172.21.15.32:0/475373212 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 msgr2=0x7f20480c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 --2- 172.21.15.32:0/475373212 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480c9270 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f203c0097f0 tx=0x7f203c01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 -- 172.21.15.32:0/475373212 shutdown_connections 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 --2- 172.21.15.32:0/475373212 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480c9270 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 -- 172.21.15.32:0/475373212 >> 172.21.15.32:0/475373212 conn(0x7f2048070f90 msgr2=0x7f20480713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 -- 172.21.15.32:0/475373212 shutdown_connections 2024-03-16T20:08:25.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.224+0000 7f204f00a640 1 -- 172.21.15.32:0/475373212 wait complete. 2024-03-16T20:08:25.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 Processor -- start 2024-03-16T20:08:25.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 -- start start 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20480c97b0 0x7f20481f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204815bde0 con 0x7f204815b2a0 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204814c7b0 con 0x7f20480c97b0 2024-03-16T20:08:25.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204f00a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20481ede90 con 0x7f2048151ef0 2024-03-16T20:08:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204d5a7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204d5a7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36772/0 (socket says 172.21.15.32:36772) 2024-03-16T20:08:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204d5a7640 1 -- 172.21.15.32:0/559510793 learned_addr learned my addr 172.21.15.32:0/559510793 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f204cda6640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20480c97b0 0x7f20481f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.228+0000 7f2047fff640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.232+0000 7f204d5a7640 1 -- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 msgr2=0x7f20480b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.232+0000 7f204d5a7640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.232+0000 7f204d5a7640 1 -- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20480c97b0 msgr2=0x7f20481f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.232+0000 7f204d5a7640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20480c97b0 0x7f20481f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.232+0000 7f204d5a7640 1 -- 172.21.15.32:0/559510793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c009420 con 0x7f204815b2a0 2024-03-16T20:08:25.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.236+0000 7f204d5a7640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f204000c730 tx=0x7f204000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:25.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.240+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040003b80 con 0x7f204815b2a0 2024-03-16T20:08:25.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.240+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040003ce0 con 0x7f204815b2a0 2024-03-16T20:08:25.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.240+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204000aa30 con 0x7f204815b2a0 2024-03-16T20:08:25.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.240+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20481f19c0 con 0x7f204815b2a0 2024-03-16T20:08:25.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.240+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20480b22f0 con 0x7f204815b2a0 2024-03-16T20:08:25.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.244+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2048068000 con 0x7f204815b2a0 2024-03-16T20:08:25.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.264+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f204000ab90 con 0x7f204815b2a0 2024-03-16T20:08:25.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.268+0000 7f20457fa640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 0x7f202c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:25.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.268+0000 7f204cda6640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 0x7f202c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:25.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.268+0000 7f204cda6640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 0x7f202c069300 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f203c0b2000 tx=0x7f203c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:25.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.544+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f204000ae40 con 0x7f204815b2a0 2024-03-16T20:08:25.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.544+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2040052020 con 0x7f204815b2a0 2024-03-16T20:08:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.756+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2048151720 con 0x7f204815b2a0 2024-03-16T20:08:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.776+0000 7f20457fa640 1 -- 172.21.15.32:0/559510793 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f20400556e0 con 0x7f204815b2a0 2024-03-16T20:08:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 msgr2=0x7f202c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 0x7f202c069300 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f203c0b2000 tx=0x7f203c008040 comp rx=0 tx=0).stop 2024-03-16T20:08:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 msgr2=0x7f204814a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f204000c730 tx=0x7f204000cbf0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 shutdown_connections 2024-03-16T20:08:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f202c066e40 0x7f202c069300 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f204815b2a0 0x7f204814a000 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2048151ef0 0x7f20480b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 --2- 172.21.15.32:0/559510793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20480c97b0 0x7f20481f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 >> 172.21.15.32:0/559510793 conn(0x7f2048070f90 msgr2=0x7f204814b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 shutdown_connections 2024-03-16T20:08:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:25.780+0000 7f204f00a640 1 -- 172.21.15.32:0/559510793 wait complete. 2024-03-16T20:08:26.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:26.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:27.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 Processor -- start 2024-03-16T20:08:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 -- start start 2024-03-16T20:08:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7dc153fa0 0x7fa7dc070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7dc0c7470 0x7fa7dc15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc0c8590 con 0x7fa7dc15d920 2024-03-16T20:08:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc0c3f00 con 0x7fa7dc0c7470 2024-03-16T20:08:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e306a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc070650 con 0x7fa7dc153fa0 2024-03-16T20:08:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7dbfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7dc0c7470 0x7fa7dc15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e1607640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7dc15d920 0x7fa7dc0724a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54546/0 (socket says 172.21.15.32:54546) 2024-03-16T20:08:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e1607640 1 -- 172.21.15.32:0/1140599328 learned_addr learned my addr 172.21.15.32:0/1140599328 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.528+0000 7fa7e0e06640 1 --2- 172.21.15.32:0/1140599328 >> v2:172.21.15.59:3300/0 conn(0x7fa7dc153fa0 0x7fa7dc070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.532+0000 7fa7dbfff640 1 -- 172.21.15.32:0/1140599328 >> v2:172.21.15.59:3300/0 conn(0x7fa7dc153fa0 msgr2=0x7fa7dc070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.532+0000 7fa7dbfff640 1 --2- 172.21.15.32:0/1140599328 >> v2:172.21.15.59:3300/0 conn(0x7fa7dc153fa0 0x7fa7dc070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.532+0000 7fa7dbfff640 1 -- 172.21.15.32:0/1140599328 >> v1:172.21.15.32:6789/0 conn(0x7fa7dc15d920 legacy=0x7fa7dc0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.532+0000 7fa7dbfff640 1 -- 172.21.15.32:0/1140599328 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc073590 con 0x7fa7dc0c7470 2024-03-16T20:08:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.536+0000 7fa7dbfff640 1 --2- 172.21.15.32:0/1140599328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15d3e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc070230 tx=0x7fa7c801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f6388fd61f7be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:27.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7db7fe640 1 -- 172.21.15.32:0/1140599328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80a8930 con 0x7fa7dc0c7470 2024-03-16T20:08:27.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7db7fe640 1 -- 172.21.15.32:0/1140599328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c80a8a90 con 0x7fa7dc0c7470 2024-03-16T20:08:27.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7db7fe640 1 -- 172.21.15.32:0/1140599328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80a8e20 con 0x7fa7dc0c7470 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 -- 172.21.15.32:0/1140599328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc0c7470 msgr2=0x7fa7dc15d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 --2- 172.21.15.32:0/1140599328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15d3e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc070230 tx=0x7fa7c801ec30 comp rx=0 tx=0).stop 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 -- 172.21.15.32:0/1140599328 shutdown_connections 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 --2- 172.21.15.32:0/1140599328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15d3e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 --2- 172.21.15.32:0/1140599328 >> v2:172.21.15.59:3300/0 conn(0x7fa7dc153fa0 0x7fa7dc070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 -- 172.21.15.32:0/1140599328 >> 172.21.15.32:0/1140599328 conn(0x7fa7dc06c0d0 msgr2=0x7fa7dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 -- 172.21.15.32:0/1140599328 shutdown_connections 2024-03-16T20:08:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.540+0000 7fa7e306a640 1 -- 172.21.15.32:0/1140599328 wait complete. 2024-03-16T20:08:27.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 Processor -- start 2024-03-16T20:08:27.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 -- start start 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7dc153fa0 0x7fa7dc14cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc14c850 con 0x7fa7dc15d920 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc14bf60 con 0x7fa7dc0c7470 2024-03-16T20:08:27.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e306a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0703d0 con 0x7fa7dc153fa0 2024-03-16T20:08:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e1607640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7dc153fa0 0x7fa7dc14cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e1607640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36792/0 (socket says 172.21.15.32:36792) 2024-03-16T20:08:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e1607640 1 -- 172.21.15.32:0/419288573 learned_addr learned my addr 172.21.15.32:0/419288573 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.544+0000 7fa7e0e06640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 -- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7dc153fa0 msgr2=0x7fa7dc14cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7dc153fa0 0x7fa7dc14cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 -- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7dc0c7470 msgr2=0x7fa7dc15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 -- 172.21.15.32:0/419288573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c8009420 con 0x7fa7dc15d920 2024-03-16T20:08:27.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.552+0000 7fa7e1607640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400cb20 tx=0x7fa7d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400aca0 con 0x7fa7dc15d920 2024-03-16T20:08:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d400ae20 con 0x7fa7dc15d920 2024-03-16T20:08:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4012640 con 0x7fa7dc15d920 2024-03-16T20:08:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc1f29b0 con 0x7fa7dc15d920 2024-03-16T20:08:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc14c470 con 0x7fa7dc15d920 2024-03-16T20:08:27.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.560+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc14c990 con 0x7fa7dc15d920 2024-03-16T20:08:27.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.592+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa7d40127a0 con 0x7fa7dc15d920 2024-03-16T20:08:27.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.592+0000 7fa7d97fa640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 0x7fa7c0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:27.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.592+0000 7fa7e0e06640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 0x7fa7c0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:27.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.592+0000 7fa7e0e06640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 0x7fa7c0069330 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009b60 tx=0x7fa7d0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:27.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.596+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa7d40563f0 con 0x7fa7dc15d920 2024-03-16T20:08:27.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:27.884+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa7d4052690 con 0x7fa7dc15d920 2024-03-16T20:08:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.088+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7dc14c990 con 0x7fa7dc15d920 2024-03-16T20:08:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.100+0000 7fa7d97fa640 1 -- 172.21.15.32:0/419288573 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fa7dc14c990 con 0x7fa7dc15d920 2024-03-16T20:08:28.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:28.091 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 msgr2=0x7fa7c0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 0x7fa7c0069330 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009b60 tx=0x7fa7d0009190 comp rx=0 tx=0).stop 2024-03-16T20:08:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 msgr2=0x7fa7dc14d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400cb20 tx=0x7fa7d400a060 comp rx=0 tx=0).stop 2024-03-16T20:08:28.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 shutdown_connections 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa7c0066e70 0x7fa7c0069330 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7dc15d920 0x7fa7dc14d1b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa7dc153fa0 0x7fa7dc14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 --2- 172.21.15.32:0/419288573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7dc0c7470 0x7fa7dc15ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 >> 172.21.15.32:0/419288573 conn(0x7fa7dc06c0d0 msgr2=0x7fa7dc0c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 shutdown_connections 2024-03-16T20:08:28.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:28.104+0000 7fa7e306a640 1 -- 172.21.15.32:0/419288573 wait complete. 2024-03-16T20:08:28.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:28.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:29.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:29.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 Processor -- start 2024-03-16T20:08:29.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 -- start start 2024-03-16T20:08:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd0151000 0x7fadd00c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadd00c36b0 0x7fadd015a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00c8ec0 con 0x7fadd00c8980 2024-03-16T20:08:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd0072f10 con 0x7fadd00c36b0 2024-03-16T20:08:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd7776640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00c5000 con 0x7fadd0151000 2024-03-16T20:08:29.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd4d11640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadd00c36b0 0x7fadd015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd5512640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd0151000 0x7fadd00c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd4d11640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadd00c36b0 0x7fadd015a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43634/0 (socket says 172.21.15.32:43634) 2024-03-16T20:08:29.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.840+0000 7fadd4d11640 1 -- 172.21.15.32:0/2644580894 learned_addr learned my addr 172.21.15.32:0/2644580894 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:29.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.844+0000 7fadd4d11640 1 -- 172.21.15.32:0/2644580894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd0151000 msgr2=0x7fadd00c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:29.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.844+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2644580894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd0151000 0x7fadd00c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:29.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.844+0000 7fadd4d11640 1 -- 172.21.15.32:0/2644580894 >> v1:172.21.15.32:6789/0 conn(0x7fadd00c8980 legacy=0x7fadd00c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:29.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.844+0000 7fadd4d11640 1 -- 172.21.15.32:0/2644580894 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd0072950 con 0x7fadd00c36b0 2024-03-16T20:08:29.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.852+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2644580894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c36b0 0x7fadd015a240 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa1173ab3745a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadc7fff640 1 -- 172.21.15.32:0/2644580894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc00a9620 con 0x7fadd00c36b0 2024-03-16T20:08:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadc7fff640 1 -- 172.21.15.32:0/2644580894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc00a9780 con 0x7fadd00c36b0 2024-03-16T20:08:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadc7fff640 1 -- 172.21.15.32:0/2644580894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc00a9b10 con 0x7fadd00c36b0 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- 172.21.15.32:0/2644580894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c36b0 msgr2=0x7fadd015a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 --2- 172.21.15.32:0/2644580894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c36b0 0x7fadd015a240 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc001c560 comp rx=0 tx=0).stop 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- 172.21.15.32:0/2644580894 shutdown_connections 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 --2- 172.21.15.32:0/2644580894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c36b0 0x7fadd015a240 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 --2- 172.21.15.32:0/2644580894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd0151000 0x7fadd00c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- 172.21.15.32:0/2644580894 >> 172.21.15.32:0/2644580894 conn(0x7fadd006c0d0 msgr2=0x7fadd006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- 172.21.15.32:0/2644580894 shutdown_connections 2024-03-16T20:08:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- 172.21.15.32:0/2644580894 wait complete. 2024-03-16T20:08:29.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 Processor -- start 2024-03-16T20:08:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.860+0000 7fadd7776640 1 -- start start 2024-03-16T20:08:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 0x7fadd01404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 0x7fadd0159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd0151000 0x7fadd01413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00ca660 con 0x7fadd00c8980 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00cbce0 con 0x7fadd0151000 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd7776640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0140d30 con 0x7fadd00c36b0 2024-03-16T20:08:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd5512640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 0x7fadd01404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd5512640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 0x7fadd01404a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:49174/0 (socket says 172.21.15.32:49174) 2024-03-16T20:08:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd5512640 1 -- 172.21.15.32:0/2608941480 learned_addr learned my addr 172.21.15.32:0/2608941480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 0x7fadd0159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.864+0000 7fadd5d13640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd0151000 0x7fadd01413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.868+0000 7fadd4d11640 1 -- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 msgr2=0x7fadd01404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.868+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 0x7fadd01404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.868+0000 7fadd4d11640 1 -- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd0151000 msgr2=0x7fadd01413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.868+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd0151000 0x7fadd01413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.868+0000 7fadd4d11640 1 -- 172.21.15.32:0/2608941480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00094d0 con 0x7fadd00c8980 2024-03-16T20:08:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.872+0000 7fadd4d11640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 0x7fadd0159420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fadc00aa000 tx=0x7fadc001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.876+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc00bc450 con 0x7fadd00c8980 2024-03-16T20:08:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.876+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc00bc5b0 con 0x7fadd00c8980 2024-03-16T20:08:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.876+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc00bc960 con 0x7fadd00c8980 2024-03-16T20:08:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.876+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd00cbf80 con 0x7fadd00c8980 2024-03-16T20:08:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.876+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd00cc540 con 0x7fadd00c8980 2024-03-16T20:08:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.880+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd00cc730 con 0x7fadd00c8980 2024-03-16T20:08:29.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.900+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fadc00c53f0 con 0x7fadd00c8980 2024-03-16T20:08:29.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.900+0000 7fadc5ffb640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 0x7fadb4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:29.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.900+0000 7fadd5512640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 0x7fadb4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:29.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:29.900+0000 7fadd5512640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 0x7fadb4069350 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009b80 tx=0x7fadcc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:30.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.192+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fadc00f01f0 con 0x7fadd00c8980 2024-03-16T20:08:30.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.192+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fadc00ec490 con 0x7fadd00c8980 2024-03-16T20:08:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.388+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd0068000 con 0x7fadd00c8980 2024-03-16T20:08:30.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.412+0000 7fadc5ffb640 1 -- 172.21.15.32:0/2608941480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fadc00efb50 con 0x7fadd00c8980 2024-03-16T20:08:30.401 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:30.401 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:30.401 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 msgr2=0x7fadb4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 0x7fadb4069350 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009b80 tx=0x7fadcc009190 comp rx=0 tx=0).stop 2024-03-16T20:08:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 msgr2=0x7fadd0159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 0x7fadd0159420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fadc00aa000 tx=0x7fadc001ada0 comp rx=0 tx=0).stop 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 shutdown_connections 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fadb4066e90 0x7fadb4069350 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd0151000 0x7fadd01413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadd00c8980 0x7fadd0159420 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 --2- 172.21.15.32:0/2608941480 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00c36b0 0x7fadd01404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 >> 172.21.15.32:0/2608941480 conn(0x7fadd006c0d0 msgr2=0x7fadd00c5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 shutdown_connections 2024-03-16T20:08:30.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:30.416+0000 7fadd7776640 1 -- 172.21.15.32:0/2608941480 wait complete. 2024-03-16T20:08:31.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:31.026 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:32.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:32.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 Processor -- start 2024-03-16T20:08:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 -- start start 2024-03-16T20:08:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ed4154350 0x7f2ed406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ed4072a10 0x7f2ed4072e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed406bae0 con 0x7f2ed40c2d70 2024-03-16T20:08:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed415a540 con 0x7f2ed4072a10 2024-03-16T20:08:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2edba1d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40c7fd0 con 0x7f2ed4154350 2024-03-16T20:08:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2ed9fba640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ed4072a10 0x7f2ed4072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2ed8fb8640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2ed40c2d70 0x7f2ed415a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42534/0 (socket says 172.21.15.32:42534) 2024-03-16T20:08:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2ed8fb8640 1 -- 172.21.15.32:0/217908212 learned_addr learned my addr 172.21.15.32:0/217908212 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.148+0000 7f2ed97b9640 1 --2- 172.21.15.32:0/217908212 >> v2:172.21.15.59:3300/0 conn(0x7f2ed4154350 0x7f2ed406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.152+0000 7f2ed9fba640 1 -- 172.21.15.32:0/217908212 >> v2:172.21.15.59:3300/0 conn(0x7f2ed4154350 msgr2=0x7f2ed406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:32.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.152+0000 7f2ed9fba640 1 --2- 172.21.15.32:0/217908212 >> v2:172.21.15.59:3300/0 conn(0x7f2ed4154350 0x7f2ed406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.152+0000 7f2ed9fba640 1 -- 172.21.15.32:0/217908212 >> v1:172.21.15.32:6789/0 conn(0x7f2ed40c2d70 legacy=0x7f2ed415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.152+0000 7f2ed9fba640 1 -- 172.21.15.32:0/217908212 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed406bd10 con 0x7f2ed4072a10 2024-03-16T20:08:32.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.160+0000 7f2ed9fba640 1 --2- 172.21.15.32:0/217908212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed4072e60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40c9a80 tx=0x7f2ecc01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b2541fa898c46f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:32.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2ecbfff640 1 -- 172.21.15.32:0/217908212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0a5850 con 0x7f2ed4072a10 2024-03-16T20:08:32.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2ecbfff640 1 -- 172.21.15.32:0/217908212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc0a59b0 con 0x7f2ed4072a10 2024-03-16T20:08:32.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2ecbfff640 1 -- 172.21.15.32:0/217908212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0a5d40 con 0x7f2ed4072a10 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 -- 172.21.15.32:0/217908212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 msgr2=0x7f2ed4072e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 --2- 172.21.15.32:0/217908212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed4072e60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40c9a80 tx=0x7f2ecc01c8a0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 -- 172.21.15.32:0/217908212 shutdown_connections 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 --2- 172.21.15.32:0/217908212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed4072e60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 --2- 172.21.15.32:0/217908212 >> v2:172.21.15.59:3300/0 conn(0x7f2ed4154350 0x7f2ed406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 -- 172.21.15.32:0/217908212 >> 172.21.15.32:0/217908212 conn(0x7f2ed406afc0 msgr2=0x7f2ed406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:32.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 -- 172.21.15.32:0/217908212 shutdown_connections 2024-03-16T20:08:32.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.164+0000 7f2edba1d640 1 -- 172.21.15.32:0/217908212 wait complete. 2024-03-16T20:08:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 Processor -- start 2024-03-16T20:08:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 -- start start 2024-03-16T20:08:32.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ed40c2d70 0x7f2ed4143b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ed4154350 0x7f2ed40caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41f5bf0 con 0x7f2ed4072a10 2024-03-16T20:08:32.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41f6770 con 0x7f2ed4154350 2024-03-16T20:08:32.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2edba1d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41440d0 con 0x7f2ed40c2d70 2024-03-16T20:08:32.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2ed97b9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2ed97b9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43646/0 (socket says 172.21.15.32:43646) 2024-03-16T20:08:32.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2ed97b9640 1 -- 172.21.15.32:0/2107966269 learned_addr learned my addr 172.21.15.32:0/2107966269 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:32.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2ed8fb8640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ed40c2d70 0x7f2ed4143b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.168+0000 7f2ed9fba640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ed4154350 0x7f2ed40caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.176+0000 7f2ed97b9640 1 -- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ed40c2d70 msgr2=0x7f2ed4143b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.176+0000 7f2ed97b9640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ed40c2d70 0x7f2ed4143b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.176+0000 7f2ed97b9640 1 -- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ed4154350 msgr2=0x7f2ed40caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.176+0000 7f2ed97b9640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ed4154350 0x7f2ed40caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.176+0000 7f2ed97b9640 1 -- 172.21.15.32:0/2107966269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc009a70 con 0x7f2ed4072a10 2024-03-16T20:08:32.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.180+0000 7f2ed97b9640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e8a0 tx=0x7f2ed000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:32.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0009510 con 0x7f2ed4072a10 2024-03-16T20:08:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed000c940 con 0x7f2ed4072a10 2024-03-16T20:08:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed40cb900 con 0x7f2ed4072a10 2024-03-16T20:08:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000ccb0 con 0x7f2ed4072a10 2024-03-16T20:08:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed41f6120 con 0x7f2ed4072a10 2024-03-16T20:08:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.184+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed40b2180 con 0x7f2ed4072a10 2024-03-16T20:08:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.216+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2ed0013470 con 0x7f2ed4072a10 2024-03-16T20:08:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.216+0000 7f2ec9ffb640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 0x7f2eb8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.216+0000 7f2ed8fb8640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 0x7f2eb8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.220+0000 7f2ed8fb8640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 0x7f2eb8069350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4158d90 tx=0x7f2ec4007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.224+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2ed0055e40 con 0x7f2ed4072a10 2024-03-16T20:08:32.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.512+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f2ed00520e0 con 0x7f2ed4072a10 2024-03-16T20:08:32.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.720+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed40674e0 con 0x7f2ed4072a10 2024-03-16T20:08:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.732+0000 7f2ec9ffb640 1 -- 172.21.15.32:0/2107966269 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f2ed0021020 con 0x7f2ed4072a10 2024-03-16T20:08:32.721 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:32.722 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 msgr2=0x7f2eb8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 0x7f2eb8069350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4158d90 tx=0x7f2ec4007e90 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 msgr2=0x7f2ed40ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e8a0 tx=0x7f2ed000eca0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 shutdown_connections 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2eb8066e90 0x7f2eb8069350 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ed4154350 0x7f2ed40caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2ed40c2d70 0x7f2ed4143b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 --2- 172.21.15.32:0/2107966269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed4072a10 0x7f2ed40ca6a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 >> 172.21.15.32:0/2107966269 conn(0x7f2ed406afc0 msgr2=0x7f2ed41595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 shutdown_connections 2024-03-16T20:08:32.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:32.736+0000 7f2edba1d640 1 -- 172.21.15.32:0/2107966269 wait complete. 2024-03-16T20:08:33.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:33.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:34.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:34.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.464+0000 7f08ddd58640 1 Processor -- start 2024-03-16T20:08:34.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.464+0000 7f08ddd58640 1 -- start start 2024-03-16T20:08:34.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08ddd58640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08d80c6340 0x7f08d80c2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:34.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08ddd58640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8157910 con 0x7f08d8150fc0 2024-03-16T20:08:34.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08ddd58640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80c2b50 con 0x7f08d80cacc0 2024-03-16T20:08:34.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08ddd58640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d8156d40 con 0x7f08d80c6340 2024-03-16T20:08:34.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08d6ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08d80c6340 0x7f08d80c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:34.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08d77fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f08d8150fc0 0x7f08d80c2e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42536/0 (socket says 172.21.15.32:42536) 2024-03-16T20:08:34.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.468+0000 7f08d77fe640 1 -- 172.21.15.32:0/2385945277 learned_addr learned my addr 172.21.15.32:0/2385945277 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:34.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.472+0000 7f08d6ffd640 1 -- 172.21.15.32:0/2385945277 >> v1:172.21.15.59:6789/0 conn(0x7f08d80cacc0 legacy=0x7f08d80c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:34.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.472+0000 7f08d6ffd640 1 -- 172.21.15.32:0/2385945277 >> v1:172.21.15.32:6789/0 conn(0x7f08d8150fc0 legacy=0x7f08d80c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:34.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.472+0000 7f08d6ffd640 1 -- 172.21.15.32:0/2385945277 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8156fe0 con 0x7f08d80c6340 2024-03-16T20:08:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.476+0000 7f08d6ffd640 1 --2- 172.21.15.32:0/2385945277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80c6340 0x7f08d80c2510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097c0 tx=0x7f08cc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce3aa4039f0528f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08d67fc640 1 -- 172.21.15.32:0/2385945277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0a8990 con 0x7f08d80c6340 2024-03-16T20:08:34.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08d67fc640 1 -- 172.21.15.32:0/2385945277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc0a8af0 con 0x7f08d80c6340 2024-03-16T20:08:34.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08d67fc640 1 -- 172.21.15.32:0/2385945277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0a8e80 con 0x7f08d80c6340 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 -- 172.21.15.32:0/2385945277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80c6340 msgr2=0x7f08d80c2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 --2- 172.21.15.32:0/2385945277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80c6340 0x7f08d80c2510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097c0 tx=0x7f08cc01ec90 comp rx=0 tx=0).stop 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 -- 172.21.15.32:0/2385945277 shutdown_connections 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 --2- 172.21.15.32:0/2385945277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80c6340 0x7f08d80c2510 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 -- 172.21.15.32:0/2385945277 >> 172.21.15.32:0/2385945277 conn(0x7f08d806afc0 msgr2=0x7f08d806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 -- 172.21.15.32:0/2385945277 shutdown_connections 2024-03-16T20:08:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.480+0000 7f08ddd58640 1 -- 172.21.15.32:0/2385945277 wait complete. 2024-03-16T20:08:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 Processor -- start 2024-03-16T20:08:34.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 -- start start 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08d80c6340 0x7f08d8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d80cacc0 0x7f08d8157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d815ac90 con 0x7f08d8150fc0 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80c8820 con 0x7f08d80cacc0 2024-03-16T20:08:34.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08ddd58640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8152640 con 0x7f08d80c6340 2024-03-16T20:08:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08d7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08d77fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08d80c6340 0x7f08d8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08d7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43650/0 (socket says 172.21.15.32:43650) 2024-03-16T20:08:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08d7fff640 1 -- 172.21.15.32:0/549953720 learned_addr learned my addr 172.21.15.32:0/549953720 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.484+0000 7f08d6ffd640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d80cacc0 0x7f08d8157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:34.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.492+0000 7f08d7fff640 1 -- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08d80c6340 msgr2=0x7f08d8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:34.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.492+0000 7f08d7fff640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08d80c6340 0x7f08d8151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:34.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.492+0000 7f08d7fff640 1 -- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d80cacc0 msgr2=0x7f08d8157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:34.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.492+0000 7f08d7fff640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d80cacc0 0x7f08d8157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:34.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.492+0000 7f08d7fff640 1 -- 172.21.15.32:0/549953720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc009420 con 0x7f08d8150fc0 2024-03-16T20:08:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.496+0000 7f08d7fff640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c710 tx=0x7f08d000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.496+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0006d60 con 0x7f08d8150fc0 2024-03-16T20:08:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.496+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d815a3f0 con 0x7f08d8150fc0 2024-03-16T20:08:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.496+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d815a9b0 con 0x7f08d8150fc0 2024-03-16T20:08:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.500+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d8068000 con 0x7f08d8150fc0 2024-03-16T20:08:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.780+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d000ce20 con 0x7f08d8150fc0 2024-03-16T20:08:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.780+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0003ba0 con 0x7f08d8150fc0 2024-03-16T20:08:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.780+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08d000a850 con 0x7f08d8150fc0 2024-03-16T20:08:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.780+0000 7f08bffff640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 0x7f08c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.780+0000 7f08d77fe640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 0x7f08c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:34.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.784+0000 7f08d77fe640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 0x7f08c4069300 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005890 tx=0x7f08c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:34.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.804+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f08d0055ec0 con 0x7f08d8150fc0 2024-03-16T20:08:34.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:34.804+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f08d0052160 con 0x7f08d8150fc0 2024-03-16T20:08:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.036+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08d80cb590 con 0x7f08d8150fc0 2024-03-16T20:08:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.048+0000 7f08bffff640 1 -- 172.21.15.32:0/549953720 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f08d0055820 con 0x7f08d8150fc0 2024-03-16T20:08:35.038 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:35.038 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:35.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 msgr2=0x7f08c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 0x7f08c4069300 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005890 tx=0x7f08c8009040 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 msgr2=0x7f08d80c6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c710 tx=0x7f08d000cbd0 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 shutdown_connections 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f08c4066e40 0x7f08c4069300 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d8150fc0 0x7f08d80c6e90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d80cacc0 0x7f08d8157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 --2- 172.21.15.32:0/549953720 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f08d80c6340 0x7f08d8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 >> 172.21.15.32:0/549953720 conn(0x7f08d806afc0 msgr2=0x7f08d814e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 shutdown_connections 2024-03-16T20:08:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:35.052+0000 7f08ddd58640 1 -- 172.21.15.32:0/549953720 wait complete. 2024-03-16T20:08:35.666 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:35.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:36.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.784+0000 7f8a62baa640 1 Processor -- start 2024-03-16T20:08:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.784+0000 7f8a62baa640 1 -- start start 2024-03-16T20:08:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a62baa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a62baa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c15e160 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a62baa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c0c2b20 con 0x7f8a5c0ca0b0 2024-03-16T20:08:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a62baa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c2920 con 0x7f8a5c0c2520 2024-03-16T20:08:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a61147640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a61147640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:35180/0 (socket says 172.21.15.32:35180) 2024-03-16T20:08:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.788+0000 7f8a61147640 1 -- 172.21.15.32:0/2749743749 learned_addr learned my addr 172.21.15.32:0/2749743749 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.800+0000 7f8a61147640 1 -- 172.21.15.32:0/2749743749 >> v1:172.21.15.59:6789/0 conn(0x7f8a5c0ca0b0 legacy=0x7f8a5c0c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.800+0000 7f8a61147640 1 -- 172.21.15.32:0/2749743749 >> v1:172.21.15.32:6789/0 conn(0x7f8a5c0c81e0 legacy=0x7f8a5c15a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.800+0000 7f8a61147640 1 -- 172.21.15.32:0/2749743749 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c15d550 con 0x7f8a5c0c2520 2024-03-16T20:08:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.808+0000 7f8a61147640 1 --2- 172.21.15.32:0/2749743749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009e50 tx=0x7f8a5401f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4413292afb9c33db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:36.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a5b7fe640 1 -- 172.21.15.32:0/2749743749 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a540af400 con 0x7f8a5c0c2520 2024-03-16T20:08:36.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a5b7fe640 1 -- 172.21.15.32:0/2749743749 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a540ab050 con 0x7f8a5c0c2520 2024-03-16T20:08:36.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a5b7fe640 1 -- 172.21.15.32:0/2749743749 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a540af780 con 0x7f8a5c0c2520 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- 172.21.15.32:0/2749743749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0c2520 msgr2=0x7f8a5c0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 --2- 172.21.15.32:0/2749743749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009e50 tx=0x7f8a5401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- 172.21.15.32:0/2749743749 shutdown_connections 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 --2- 172.21.15.32:0/2749743749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0c2520 0x7f8a5c0ca6e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- 172.21.15.32:0/2749743749 >> 172.21.15.32:0/2749743749 conn(0x7f8a5c06afc0 msgr2=0x7f8a5c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- 172.21.15.32:0/2749743749 shutdown_connections 2024-03-16T20:08:36.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- 172.21.15.32:0/2749743749 wait complete. 2024-03-16T20:08:36.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 Processor -- start 2024-03-16T20:08:36.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.816+0000 7f8a62baa640 1 -- start start 2024-03-16T20:08:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a5c0c2520 0x7f8a5c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0ca0b0 0x7f8a5c0c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c15cdb0 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c1535d0 con 0x7f8a5c0ca0b0 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a62baa640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c143b50 con 0x7f8a5c0c2520 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43662/0 (socket says 172.21.15.32:43662) 2024-03-16T20:08:36.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a5bfff640 1 -- 172.21.15.32:0/1945003711 learned_addr learned my addr 172.21.15.32:0/1945003711 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:36.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a60946640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a5c0c2520 0x7f8a5c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:36.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.820+0000 7f8a61147640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0ca0b0 0x7f8a5c0c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.824+0000 7f8a5bfff640 1 -- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a5c0c2520 msgr2=0x7f8a5c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.824+0000 7f8a5bfff640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a5c0c2520 0x7f8a5c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.824+0000 7f8a5bfff640 1 -- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0ca0b0 msgr2=0x7f8a5c0c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.824+0000 7f8a5bfff640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0ca0b0 0x7f8a5c0c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.824+0000 7f8a5bfff640 1 -- 172.21.15.32:0/1945003711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a54009b30 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.828+0000 7f8a5bfff640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800c370 tx=0x7f8a4800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a480052d0 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48005430 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48007260 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a62baa640 1 -- 172.21.15.32:0/1945003711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c15d200 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a62baa640 1 -- 172.21.15.32:0/1945003711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c1e6700 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.836+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a340052a0 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.868+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8a480073c0 con 0x7f8a5c0c81e0 2024-03-16T20:08:36.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.868+0000 7f8a597fa640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 0x7f8a40069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:36.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.868+0000 7f8a60946640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 0x7f8a40069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:36.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:36.868+0000 7f8a60946640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 0x7f8a40069380 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a50007fd0 tx=0x7f8a50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.156+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8a48007670 con 0x7f8a5c0c81e0 2024-03-16T20:08:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.156+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8a480510d0 con 0x7f8a5c0c81e0 2024-03-16T20:08:37.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.456+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a34005900 con 0x7f8a5c0c81e0 2024-03-16T20:08:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.476+0000 7f8a597fa640 1 -- 172.21.15.32:0/1945003711 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8a34005900 con 0x7f8a5c0c81e0 2024-03-16T20:08:37.467 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:37.467 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 msgr2=0x7f8a40069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 0x7f8a40069380 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a50007fd0 tx=0x7f8a50009040 comp rx=0 tx=0).stop 2024-03-16T20:08:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 msgr2=0x7f8a5c143610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800c370 tx=0x7f8a4800c830 comp rx=0 tx=0).stop 2024-03-16T20:08:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 shutdown_connections 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a40066ec0 0x7f8a40069380 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a5c0ca0b0 0x7f8a5c0c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c0c81e0 0x7f8a5c143610 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 --2- 172.21.15.32:0/1945003711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a5c0c2520 0x7f8a5c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 >> 172.21.15.32:0/1945003711 conn(0x7f8a5c06afc0 msgr2=0x7f8a5c0cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 shutdown_connections 2024-03-16T20:08:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:37.480+0000 7f8a46ffd640 1 -- 172.21.15.32:0/1945003711 wait complete. 2024-03-16T20:08:38.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:38.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:39.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:39.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 Processor -- start 2024-03-16T20:08:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 -- start start 2024-03-16T20:08:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6420 0x7f7e4c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e4c158390 0x7f7e4c1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c070760 con 0x7f7e4c0c6040 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0731c0 con 0x7f7e4c158390 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.220+0000 7f7e53aa4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0734a0 con 0x7f7e4c0c6420 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.224+0000 7f7e52041640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e4c158390 0x7f7e4c1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.224+0000 7f7e5103f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6420 0x7f7e4c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.224+0000 7f7e52041640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e4c158390 0x7f7e4c1516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:36044/0 (socket says 172.21.15.32:36044) 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.224+0000 7f7e52041640 1 -- 172.21.15.32:0/2394627282 learned_addr learned my addr 172.21.15.32:0/2394627282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.228+0000 7f7e52041640 1 -- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6420 msgr2=0x7f7e4c0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.228+0000 7f7e52041640 1 --2- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6420 0x7f7e4c0c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.228+0000 7f7e52041640 1 -- 172.21.15.32:0/2394627282 >> v1:172.21.15.59:6789/0 conn(0x7f7e4c0c6040 legacy=0x7f7e4c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.228+0000 7f7e52041640 1 -- 172.21.15.32:0/2394627282 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c06fe30 con 0x7f7e4c158390 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.240+0000 7f7e52041640 1 --2- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1516d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009e80 tx=0x7f7e4401c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e50d62e210e238 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.244+0000 7f7e5083e640 1 -- 172.21.15.32:0/2394627282 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440a58a0 con 0x7f7e4c158390 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e5083e640 1 -- 172.21.15.32:0/2394627282 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e440a5a00 con 0x7f7e4c158390 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e5083e640 1 -- 172.21.15.32:0/2394627282 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440a5d90 con 0x7f7e4c158390 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 -- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 msgr2=0x7f7e4c1516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 --2- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1516d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009e80 tx=0x7f7e4401c9e0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 -- 172.21.15.32:0/2394627282 shutdown_connections 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 --2- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1516d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 --2- 172.21.15.32:0/2394627282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6420 0x7f7e4c0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 -- 172.21.15.32:0/2394627282 >> 172.21.15.32:0/2394627282 conn(0x7f7e4c06c0d0 msgr2=0x7f7e4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 -- 172.21.15.32:0/2394627282 shutdown_connections 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.248+0000 7f7e53aa4640 1 -- 172.21.15.32:0/2394627282 wait complete. 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 Processor -- start 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 -- start start 2024-03-16T20:08:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6040 0x7f7e4c0c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c146770 con 0x7f7e4c0c6420 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0c4850 con 0x7f7e4c158390 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e53aa4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c13f840 con 0x7f7e4c0c6040 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e5103f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e51840640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6040 0x7f7e4c0c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e5103f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58322/0 (socket says 172.21.15.32:58322) 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e5103f640 1 -- 172.21.15.32:0/3047268040 learned_addr learned my addr 172.21.15.32:0/3047268040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.252+0000 7f7e52041640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.260+0000 7f7e5103f640 1 -- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6040 msgr2=0x7f7e4c0c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.260+0000 7f7e5103f640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6040 0x7f7e4c0c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.260+0000 7f7e5103f640 1 -- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 msgr2=0x7f7e4c1f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.260+0000 7f7e5103f640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.260+0000 7f7e5103f640 1 -- 172.21.15.32:0/3047268040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e44009b30 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.264+0000 7f7e5103f640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e900 tx=0x7f7e3c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c009690 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00c9a0 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e53aa4640 1 -- 172.21.15.32:0/3047268040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c145b60 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00cd50 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e53aa4640 1 -- 172.21.15.32:0/3047268040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c146120 con 0x7f7e4c0c6420 2024-03-16T20:08:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.268+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e280052a0 con 0x7f7e4c0c6420 2024-03-16T20:08:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.580+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7e3c013470 con 0x7f7e4c0c6420 2024-03-16T20:08:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.580+0000 7f7e427fc640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 0x7f7e30069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.580+0000 7f7e51840640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 0x7f7e30069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.580+0000 7f7e51840640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 0x7f7e30069330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7e48001810 tx=0x7f7e48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.596+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7e3c055f60 con 0x7f7e4c0c6420 2024-03-16T20:08:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.596+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7e3c052200 con 0x7f7e4c0c6420 2024-03-16T20:08:39.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.800+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e28005900 con 0x7f7e4c0c6420 2024-03-16T20:08:39.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.880+0000 7f7e427fc640 1 -- 172.21.15.32:0/3047268040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f7e28005900 con 0x7f7e4c0c6420 2024-03-16T20:08:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:39.869 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:39.869 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 msgr2=0x7f7e30069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 0x7f7e30069330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7e48001810 tx=0x7f7e48009040 comp rx=0 tx=0).stop 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 msgr2=0x7f7e4c144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e900 tx=0x7f7e3c00edc0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 shutdown_connections 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7e30066e70 0x7f7e30069330 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e4c158390 0x7f7e4c1f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4c0c6420 0x7f7e4c144df0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 --2- 172.21.15.32:0/3047268040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7e4c0c6040 0x7f7e4c0c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:39.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 >> 172.21.15.32:0/3047268040 conn(0x7f7e4c06c0d0 msgr2=0x7f7e4c155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:39.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 shutdown_connections 2024-03-16T20:08:39.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:39.884+0000 7f7e2ffff640 1 -- 172.21.15.32:0/3047268040 wait complete. 2024-03-16T20:08:40.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:40.497 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:41.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 Processor -- start 2024-03-16T20:08:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 -- start start 2024-03-16T20:08:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c0073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c0151020 con 0x7f18c0150c40 2024-03-16T20:08:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c0072da0 con 0x7f18c015a420 2024-03-16T20:08:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18c5419640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c015a000 con 0x7f18c00725e0 2024-03-16T20:08:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18beffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f18c0150c40 0x7f18c0070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43626/0 (socket says 172.21.15.32:43626) 2024-03-16T20:08:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.628+0000 7f18beffd640 1 -- 172.21.15.32:0/4146894704 learned_addr learned my addr 172.21.15.32:0/4146894704 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:41.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.632+0000 7f18b7fff640 1 --2- 172.21.15.32:0/4146894704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c0073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.632+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553543718 0 0) 0x7f18c0151020 con 0x7f18c0150c40 2024-03-16T20:08:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.632+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac003530 con 0x7f18c0150c40 2024-03-16T20:08:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.636+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204533879 0 0) 0x7f18c0072da0 con 0x7f18c015a420 2024-03-16T20:08:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.636+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c0151020 con 0x7f18c015a420 2024-03-16T20:08:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.636+0000 7f18b7fff640 1 -- 172.21.15.32:0/4146894704 >> v1:172.21.15.59:6789/0 conn(0x7f18c015a420 legacy=0x7f18c0164770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:41.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.636+0000 7f18b7fff640 1 -- 172.21.15.32:0/4146894704 >> v1:172.21.15.32:6789/0 conn(0x7f18c0150c40 legacy=0x7f18c0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:41.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.636+0000 7f18b7fff640 1 -- 172.21.15.32:0/4146894704 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c015e310 con 0x7f18c00725e0 2024-03-16T20:08:41.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.644+0000 7f18b7fff640 1 --2- 172.21.15.32:0/4146894704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c0073070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097f0 tx=0x7f18a801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52ab44920999e399 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.652+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80aa640 con 0x7f18c00725e0 2024-03-16T20:08:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.652+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a80aa7a0 con 0x7f18c00725e0 2024-03-16T20:08:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.652+0000 7f18be7fc640 1 -- 172.21.15.32:0/4146894704 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80aab30 con 0x7f18c00725e0 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- 172.21.15.32:0/4146894704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 msgr2=0x7f18c0073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 --2- 172.21.15.32:0/4146894704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c0073070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097f0 tx=0x7f18a801c4e0 comp rx=0 tx=0).stop 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- 172.21.15.32:0/4146894704 shutdown_connections 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 --2- 172.21.15.32:0/4146894704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c0073070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- 172.21.15.32:0/4146894704 >> 172.21.15.32:0/4146894704 conn(0x7f18c006c0d0 msgr2=0x7f18c006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- 172.21.15.32:0/4146894704 shutdown_connections 2024-03-16T20:08:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- 172.21.15.32:0/4146894704 wait complete. 2024-03-16T20:08:41.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 Processor -- start 2024-03-16T20:08:41.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- start start 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0150c40 0x7f18c01494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c0149010 con 0x7f18c015a420 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c0146d60 con 0x7f18c0150c40 2024-03-16T20:08:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.656+0000 7f18c5419640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c0148400 con 0x7f18c00725e0 2024-03-16T20:08:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.660+0000 7f18bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.660+0000 7f18bf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58328/0 (socket says 172.21.15.32:58328) 2024-03-16T20:08:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.660+0000 7f18bf7fe640 1 -- 172.21.15.32:0/1268555347 learned_addr learned my addr 172.21.15.32:0/1268555347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.660+0000 7f18b7fff640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0150c40 0x7f18c01494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.660+0000 7f18beffd640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 -- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 msgr2=0x7f18c015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 -- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0150c40 msgr2=0x7f18c01494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0150c40 0x7f18c01494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 -- 172.21.15.32:0/1268555347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8009420 con 0x7f18c015a420 2024-03-16T20:08:41.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.664+0000 7f18bf7fe640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f18b800c730 tx=0x7f18b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.668+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8003b80 con 0x7f18c015a420 2024-03-16T20:08:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.668+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b8003ce0 con 0x7f18c015a420 2024-03-16T20:08:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.668+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b800aa90 con 0x7f18c015a420 2024-03-16T20:08:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.672+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c015d560 con 0x7f18c015a420 2024-03-16T20:08:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.672+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c01489c0 con 0x7f18c015a420 2024-03-16T20:08:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.672+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c0149180 con 0x7f18c015a420 2024-03-16T20:08:41.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.696+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f18b800abf0 con 0x7f18c015a420 2024-03-16T20:08:41.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.696+0000 7f18b77fe640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:41.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.696+0000 7f18beffd640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:41.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.696+0000 7f18beffd640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 0x7f18a0069300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005890 tx=0x7f18b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:41.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.704+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f18b800ce40 con 0x7f18c015a420 2024-03-16T20:08:41.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:41.976+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f18b8052110 con 0x7f18c015a420 2024-03-16T20:08:42.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.172+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c015e010 con 0x7f18c015a420 2024-03-16T20:08:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.188+0000 7f18b77fe640 1 -- 172.21.15.32:0/1268555347 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f18c015e010 con 0x7f18c015a420 2024-03-16T20:08:42.179 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:42.179 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 msgr2=0x7f18a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 0x7f18a0069300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005890 tx=0x7f18b0009040 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 msgr2=0x7f18c01452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f18b800c730 tx=0x7f18b800cbf0 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 shutdown_connections 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18c015a420 0x7f18c01452c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0150c40 0x7f18c01494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 --2- 172.21.15.32:0/1268555347 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f18c00725e0 0x7f18c015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 >> 172.21.15.32:0/1268555347 conn(0x7f18c006c0d0 msgr2=0x7f18c014e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 shutdown_connections 2024-03-16T20:08:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:42.192+0000 7f18c5419640 1 -- 172.21.15.32:0/1268555347 wait complete. 2024-03-16T20:08:42.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:08:42.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:08:42.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:42.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:43.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 Processor -- start 2024-03-16T20:08:43.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 -- start start 2024-03-16T20:08:43.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe03c0c81e0 0x7fe03c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:43.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c06c460 con 0x7fe03c1540c0 2024-03-16T20:08:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03c0c3f40 con 0x7fe03c0c81e0 2024-03-16T20:08:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe044056640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c15a240 con 0x7fe03c0c3570 2024-03-16T20:08:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe0415f1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe03c0c81e0 0x7fe03c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe041df2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe03c1540c0 0x7fe03c15a6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43632/0 (socket says 172.21.15.32:43632) 2024-03-16T20:08:43.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe041df2640 1 -- 172.21.15.32:0/324266097 learned_addr learned my addr 172.21.15.32:0/324266097 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:43.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.924+0000 7fe0425f3640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:43.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe040df0640 1 -- 172.21.15.32:0/324266097 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533547858 0 0) 0x7fe03c06c460 con 0x7fe03c1540c0 2024-03-16T20:08:43.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe040df0640 1 -- 172.21.15.32:0/324266097 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe028003530 con 0x7fe03c1540c0 2024-03-16T20:08:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe0415f1640 1 -- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 msgr2=0x7fe03c0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe0415f1640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c0c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe0415f1640 1 -- 172.21.15.32:0/324266097 >> v1:172.21.15.32:6789/0 conn(0x7fe03c1540c0 legacy=0x7fe03c15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.928+0000 7fe0415f1640 1 -- 172.21.15.32:0/324266097 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c06cf20 con 0x7fe03c0c81e0 2024-03-16T20:08:43.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.932+0000 7fe0415f1640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c0c81e0 0x7fe03c0c8630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0097c0 tx=0x7fe02c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b28b978b0b9ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe040df0640 1 -- 172.21.15.32:0/324266097 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0ab670 con 0x7fe03c0c81e0 2024-03-16T20:08:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe040df0640 1 -- 172.21.15.32:0/324266097 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c0ab7d0 con 0x7fe03c0c81e0 2024-03-16T20:08:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe040df0640 1 -- 172.21.15.32:0/324266097 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0abb60 con 0x7fe03c0c81e0 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c0c81e0 msgr2=0x7fe03c0c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c0c81e0 0x7fe03c0c8630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0097c0 tx=0x7fe02c01c470 comp rx=0 tx=0).stop 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- 172.21.15.32:0/324266097 shutdown_connections 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c0c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- 172.21.15.32:0/324266097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c0c81e0 0x7fe03c0c8630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- 172.21.15.32:0/324266097 >> 172.21.15.32:0/324266097 conn(0x7fe03c070f90 msgr2=0x7fe03c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- 172.21.15.32:0/324266097 shutdown_connections 2024-03-16T20:08:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- 172.21.15.32:0/324266097 wait complete. 2024-03-16T20:08:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 Processor -- start 2024-03-16T20:08:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- start start 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c1540c0 0x7fe03c1f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c14e320 con 0x7fe03c0c81e0 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c14dd10 con 0x7fe03c1540c0 2024-03-16T20:08:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.940+0000 7fe044056640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c0ccf20 con 0x7fe03c0c3570 2024-03-16T20:08:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.944+0000 7fe0415f1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.944+0000 7fe0415f1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58334/0 (socket says 172.21.15.32:58334) 2024-03-16T20:08:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.944+0000 7fe0415f1640 1 -- 172.21.15.32:0/4032419629 learned_addr learned my addr 172.21.15.32:0/4032419629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.944+0000 7fe041df2640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.944+0000 7fe0425f3640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c1540c0 0x7fe03c1f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.948+0000 7fe0415f1640 1 -- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 msgr2=0x7fe03c15c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.948+0000 7fe0415f1640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c15c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.948+0000 7fe0415f1640 1 -- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c1540c0 msgr2=0x7fe03c1f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.948+0000 7fe0415f1640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c1540c0 0x7fe03c1f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.948+0000 7fe0415f1640 1 -- 172.21.15.32:0/4032419629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c009420 con 0x7fe03c0c81e0 2024-03-16T20:08:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.952+0000 7fe0415f1640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0ad040 tx=0x7fe02c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0bf610 con 0x7fe03c0c81e0 2024-03-16T20:08:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c0bf770 con 0x7fe03c0c81e0 2024-03-16T20:08:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c0cd1e0 con 0x7fe03c0c81e0 2024-03-16T20:08:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0bfb20 con 0x7fe03c0c81e0 2024-03-16T20:08:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c15d110 con 0x7fe03c0c81e0 2024-03-16T20:08:43.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:43.956+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c070390 con 0x7fe03c0c81e0 2024-03-16T20:08:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.232+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe02c0bfc80 con 0x7fe03c0c81e0 2024-03-16T20:08:44.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.232+0000 7fe0327fc640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 0x7fe020069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:44.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.236+0000 7fe041df2640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 0x7fe020069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:44.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.236+0000 7fe041df2640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 0x7fe020069300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe038007fd0 tx=0x7fe038009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.252+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe02c0f0590 con 0x7fe03c0c81e0 2024-03-16T20:08:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.252+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe02c0ec830 con 0x7fe03c0c81e0 2024-03-16T20:08:44.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.456+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe03c1f2780 con 0x7fe03c0c81e0 2024-03-16T20:08:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.480+0000 7fe0327fc640 1 -- 172.21.15.32:0/4032419629 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe03c1f2780 con 0x7fe03c0c81e0 2024-03-16T20:08:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:44.469 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 msgr2=0x7fe020069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 0x7fe020069300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe038007fd0 tx=0x7fe038009040 comp rx=0 tx=0).stop 2024-03-16T20:08:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 msgr2=0x7fe03c1f2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0ad040 tx=0x7fe02c01acb0 comp rx=0 tx=0).stop 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 shutdown_connections 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe020066e40 0x7fe020069300 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe03c1540c0 0x7fe03c1f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0c81e0 0x7fe03c1f2a60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 --2- 172.21.15.32:0/4032419629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe03c0c3570 0x7fe03c15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 >> 172.21.15.32:0/4032419629 conn(0x7fe03c070f90 msgr2=0x7fe03c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 shutdown_connections 2024-03-16T20:08:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:44.484+0000 7fe044056640 1 -- 172.21.15.32:0/4032419629 wait complete. 2024-03-16T20:08:45.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:45.093 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:46.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:46.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.211+0000 7fef75240640 1 Processor -- start 2024-03-16T20:08:46.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.211+0000 7fef75240640 1 -- start start 2024-03-16T20:08:46.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef75240640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef70073050 0x7fef70150c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef75240640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef700c16a0 0x7fef70151160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef75240640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef701516a0 con 0x7fef70072c70 2024-03-16T20:08:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef75240640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef70156f20 con 0x7fef700c16a0 2024-03-16T20:08:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef75240640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7006e6a0 con 0x7fef70073050 2024-03-16T20:08:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef6f59e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef700c16a0 0x7fef70151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef6f59e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef700c16a0 0x7fef70151160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58338/0 (socket says 172.21.15.32:58338) 2024-03-16T20:08:46.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef6f59e640 1 -- 172.21.15.32:0/57048688 learned_addr learned my addr 172.21.15.32:0/57048688 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:46.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.215+0000 7fef6e59c640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef70073050 0x7fef70150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.219+0000 7fef6f59e640 1 -- 172.21.15.32:0/57048688 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef70073050 msgr2=0x7fef70150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:46.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.219+0000 7fef6f59e640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef70073050 0x7fef70150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.219+0000 7fef6f59e640 1 -- 172.21.15.32:0/57048688 >> v1:172.21.15.32:6789/0 conn(0x7fef70072c70 legacy=0x7fef700c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:46.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.219+0000 7fef6f59e640 1 -- 172.21.15.32:0/57048688 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7006c250 con 0x7fef700c16a0 2024-03-16T20:08:46.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.223+0000 7fef6f59e640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef700c16a0 0x7fef70151160 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fef68009e80 tx=0x7fef6801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c4708d0e6200ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:46.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef6dd9b640 1 -- 172.21.15.32:0/57048688 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef680a5820 con 0x7fef700c16a0 2024-03-16T20:08:46.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef6dd9b640 1 -- 172.21.15.32:0/57048688 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef680a5980 con 0x7fef700c16a0 2024-03-16T20:08:46.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef6dd9b640 1 -- 172.21.15.32:0/57048688 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef680a5d10 con 0x7fef700c16a0 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 -- 172.21.15.32:0/57048688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef700c16a0 msgr2=0x7fef70151160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef700c16a0 0x7fef70151160 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fef68009e80 tx=0x7fef6801c9a0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 -- 172.21.15.32:0/57048688 shutdown_connections 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef700c16a0 0x7fef70151160 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 --2- 172.21.15.32:0/57048688 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef70073050 0x7fef70150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 -- 172.21.15.32:0/57048688 >> 172.21.15.32:0/57048688 conn(0x7fef7006afc0 msgr2=0x7fef7006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 -- 172.21.15.32:0/57048688 shutdown_connections 2024-03-16T20:08:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.231+0000 7fef75240640 1 -- 172.21.15.32:0/57048688 wait complete. 2024-03-16T20:08:46.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 Processor -- start 2024-03-16T20:08:46.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 -- start start 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef70072c70 0x7fef701f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef70073050 0x7fef700c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef70144940 con 0x7fef70073050 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef70156000 con 0x7fef70072c70 2024-03-16T20:08:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef75240640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef70144350 con 0x7fef700c16a0 2024-03-16T20:08:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef6f59e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef6ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef70072c70 0x7fef701f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef6f59e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:53934/0 (socket says 172.21.15.32:53934) 2024-03-16T20:08:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.235+0000 7fef6f59e640 1 -- 172.21.15.32:0/3743948760 learned_addr learned my addr 172.21.15.32:0/3743948760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.239+0000 7fef6e59c640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef70073050 0x7fef700c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.243+0000 7fef6f59e640 1 -- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef70072c70 msgr2=0x7fef701f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.243+0000 7fef6f59e640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef70072c70 0x7fef701f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.243+0000 7fef6f59e640 1 -- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef70073050 msgr2=0x7fef700c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.243+0000 7fef6f59e640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef70073050 0x7fef700c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.243+0000 7fef6f59e640 1 -- 172.21.15.32:0/3743948760 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68009b30 con 0x7fef700c16a0 2024-03-16T20:08:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.247+0000 7fef6f59e640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef680af040 tx=0x7fef68019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef680b8670 con 0x7fef700c16a0 2024-03-16T20:08:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef680b87f0 con 0x7fef700c16a0 2024-03-16T20:08:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef700c7df0 con 0x7fef700c16a0 2024-03-16T20:08:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef680b8ba0 con 0x7fef700c16a0 2024-03-16T20:08:46.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef70155a40 con 0x7fef700c16a0 2024-03-16T20:08:46.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.255+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef70144a80 con 0x7fef700c16a0 2024-03-16T20:08:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.587+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef680b8d00 con 0x7fef700c16a0 2024-03-16T20:08:46.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.587+0000 7fef577fe640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 0x7fef5c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:46.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.587+0000 7fef6ed9d640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 0x7fef5c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:46.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.587+0000 7fef6ed9d640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 0x7fef5c069350 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fef60007fd0 tx=0x7fef60007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:46.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.591+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fef680f1470 con 0x7fef700c16a0 2024-03-16T20:08:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.611+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fef680ed710 con 0x7fef700c16a0 2024-03-16T20:08:46.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.835+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef70144a80 con 0x7fef700c16a0 2024-03-16T20:08:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.875+0000 7fef577fe640 1 -- 172.21.15.32:0/3743948760 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fef70144a80 con 0x7fef700c16a0 2024-03-16T20:08:46.865 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:46.865 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 msgr2=0x7fef5c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 0x7fef5c069350 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fef60007fd0 tx=0x7fef60007e40 comp rx=0 tx=0).stop 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 msgr2=0x7fef70154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef680af040 tx=0x7fef68019740 comp rx=0 tx=0).stop 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 shutdown_connections 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fef5c066e90 0x7fef5c069350 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef700c16a0 0x7fef70154d90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef70073050 0x7fef700c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 --2- 172.21.15.32:0/3743948760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef70072c70 0x7fef701f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 >> 172.21.15.32:0/3743948760 conn(0x7fef7006afc0 msgr2=0x7fef7014d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 shutdown_connections 2024-03-16T20:08:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:46.879+0000 7fef75240640 1 -- 172.21.15.32:0/3743948760 wait complete. 2024-03-16T20:08:47.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:47.489 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:48.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:48.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 Processor -- start 2024-03-16T20:08:48.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 -- start start 2024-03-16T20:08:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26a81540c0 0x7f26a815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a815ac10 0x7f26a80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a806c460 con 0x7f26a80c8720 2024-03-16T20:08:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a815a240 con 0x7f26a81540c0 2024-03-16T20:08:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.607+0000 7f26adc8d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a806cf20 con 0x7f26a815ac10 2024-03-16T20:08:48.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.611+0000 7f26a77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26a81540c0 0x7f26a815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.611+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a815ac10 0x7f26a80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.611+0000 7f26a77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26a81540c0 0x7f26a815a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:36112/0 (socket says 172.21.15.32:36112) 2024-03-16T20:08:48.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.611+0000 7f26a77fe640 1 -- 172.21.15.32:0/2960817644 learned_addr learned my addr 172.21.15.32:0/2960817644 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:48.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.615+0000 7f26a77fe640 1 -- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a815ac10 msgr2=0x7f26a80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:48.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.615+0000 7f26a77fe640 1 --2- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a815ac10 0x7f26a80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:48.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.615+0000 7f26a77fe640 1 -- 172.21.15.32:0/2960817644 >> v1:172.21.15.59:6789/0 conn(0x7f26a80c8720 legacy=0x7f26a80c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:48.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.615+0000 7f26a77fe640 1 -- 172.21.15.32:0/2960817644 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a806d200 con 0x7f26a81540c0 2024-03-16T20:08:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.623+0000 7f26a77fe640 1 --2- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a81540c0 0x7f26a815a6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2698009870 tx=0x7f269801c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afa6b98e17280e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.631+0000 7f26a67fc640 1 -- 172.21.15.32:0/2960817644 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26980a9610 con 0x7f26a81540c0 2024-03-16T20:08:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.631+0000 7f26a67fc640 1 -- 172.21.15.32:0/2960817644 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26980a9770 con 0x7f26a81540c0 2024-03-16T20:08:48.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.631+0000 7f26a67fc640 1 -- 172.21.15.32:0/2960817644 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26980a9b00 con 0x7f26a81540c0 2024-03-16T20:08:48.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a81540c0 msgr2=0x7f26a815a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:48.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a81540c0 0x7f26a815a6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2698009870 tx=0x7f269801c530 comp rx=0 tx=0).stop 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- 172.21.15.32:0/2960817644 shutdown_connections 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a815ac10 0x7f26a80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- 172.21.15.32:0/2960817644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a81540c0 0x7f26a815a6d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- 172.21.15.32:0/2960817644 >> 172.21.15.32:0/2960817644 conn(0x7f26a8070f90 msgr2=0x7f26a80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- 172.21.15.32:0/2960817644 shutdown_connections 2024-03-16T20:08:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- 172.21.15.32:0/2960817644 wait complete. 2024-03-16T20:08:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 Processor -- start 2024-03-16T20:08:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- start start 2024-03-16T20:08:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a80c8720 0x7f26a814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a815ac10 0x7f26a81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a814b430 con 0x7f26a81540c0 2024-03-16T20:08:48.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a80cd0c0 con 0x7f26a815ac10 2024-03-16T20:08:48.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.635+0000 7f26adc8d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a80ccbb0 con 0x7f26a80c8720 2024-03-16T20:08:48.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.639+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.639+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a815ac10 0x7f26a81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.639+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58348/0 (socket says 172.21.15.32:58348) 2024-03-16T20:08:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.639+0000 7f26a6ffd640 1 -- 172.21.15.32:0/3888054824 learned_addr learned my addr 172.21.15.32:0/3888054824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.639+0000 7f26a77fe640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a80c8720 0x7f26a814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 -- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a80c8720 msgr2=0x7f26a814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a80c8720 0x7f26a814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 -- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a815ac10 msgr2=0x7f26a81529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a815ac10 0x7f26a81529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 -- 172.21.15.32:0/3888054824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980094d0 con 0x7f26a81540c0 2024-03-16T20:08:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.643+0000 7f26a6ffd640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f269c00e930 tx=0x7f269c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c009690 con 0x7f26a81540c0 2024-03-16T20:08:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c00ca00 con 0x7f26a81540c0 2024-03-16T20:08:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a81f42c0 con 0x7f26a81540c0 2024-03-16T20:08:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c00cdb0 con 0x7f26a81540c0 2024-03-16T20:08:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a80ccde0 con 0x7f26a81540c0 2024-03-16T20:08:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.651+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a8068000 con 0x7f26a81540c0 2024-03-16T20:08:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.675+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f269c013470 con 0x7f26a81540c0 2024-03-16T20:08:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.675+0000 7f268ffff640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 0x7f2694069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.675+0000 7f26a77fe640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 0x7f2694069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.675+0000 7f26a77fe640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 0x7f2694069330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26980ae000 tx=0x7f26980b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.947+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f269c056080 con 0x7f26a81540c0 2024-03-16T20:08:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:48.947+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f269c052320 con 0x7f26a81540c0 2024-03-16T20:08:49.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.155+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a8152730 con 0x7f26a81540c0 2024-03-16T20:08:49.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.247+0000 7f268ffff640 1 -- 172.21.15.32:0/3888054824 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f269c021070 con 0x7f26a81540c0 2024-03-16T20:08:49.239 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:49.240 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 msgr2=0x7f2694069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 0x7f2694069330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26980ae000 tx=0x7f26980b3040 comp rx=0 tx=0).stop 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 msgr2=0x7f26a8143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f269c00e930 tx=0x7f269c00edf0 comp rx=0 tx=0).stop 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 shutdown_connections 2024-03-16T20:08:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2694066e70 0x7f2694069330 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26a815ac10 0x7f26a81529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26a81540c0 0x7f26a8143ed0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 --2- 172.21.15.32:0/3888054824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26a80c8720 0x7f26a814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 >> 172.21.15.32:0/3888054824 conn(0x7f26a8070f90 msgr2=0x7f26a8159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 shutdown_connections 2024-03-16T20:08:49.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:49.251+0000 7f26adc8d640 1 -- 172.21.15.32:0/3888054824 wait complete. 2024-03-16T20:08:49.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:49.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:50.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 Processor -- start 2024-03-16T20:08:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 -- start start 2024-03-16T20:08:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feea80c6040 0x7feea80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea8070760 con 0x7feea8158390 2024-03-16T20:08:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80731c0 con 0x7feea80c6420 2024-03-16T20:08:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feeae736640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea80734a0 con 0x7feea80c6040 2024-03-16T20:08:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.983+0000 7feea7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feea80c6040 0x7feea80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.987+0000 7feea7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feea80c6040 0x7feea80c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:34616/0 (socket says 172.21.15.32:34616) 2024-03-16T20:08:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.987+0000 7feea7fff640 1 -- 172.21.15.32:0/1105359487 learned_addr learned my addr 172.21.15.32:0/1105359487 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:50.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.991+0000 7feea7fff640 1 -- 172.21.15.32:0/1105359487 >> v1:172.21.15.59:6789/0 conn(0x7feea80c6420 legacy=0x7feea8157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:50.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.991+0000 7feea7fff640 1 -- 172.21.15.32:0/1105359487 >> v1:172.21.15.32:6789/0 conn(0x7feea8158390 legacy=0x7feea81516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:50.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.991+0000 7feea7fff640 1 -- 172.21.15.32:0/1105359487 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea806fe30 con 0x7feea80c6040 2024-03-16T20:08:50.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:50.995+0000 7feea7fff640 1 --2- 172.21.15.32:0/1105359487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6040 0x7feea80c8800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fee98009870 tx=0x7fee9801c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5f438d331ec469 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:50.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feea6ffd640 1 -- 172.21.15.32:0/1105359487 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980aa610 con 0x7feea80c6040 2024-03-16T20:08:50.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feea6ffd640 1 -- 172.21.15.32:0/1105359487 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee980a6040 con 0x7feea80c6040 2024-03-16T20:08:50.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feea6ffd640 1 -- 172.21.15.32:0/1105359487 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980aa990 con 0x7feea80c6040 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 -- 172.21.15.32:0/1105359487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6040 msgr2=0x7feea80c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 --2- 172.21.15.32:0/1105359487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6040 0x7feea80c8800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fee98009870 tx=0x7fee9801c4b0 comp rx=0 tx=0).stop 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 -- 172.21.15.32:0/1105359487 shutdown_connections 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 --2- 172.21.15.32:0/1105359487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6040 0x7feea80c8800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 -- 172.21.15.32:0/1105359487 >> 172.21.15.32:0/1105359487 conn(0x7feea806c0d0 msgr2=0x7feea806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 -- 172.21.15.32:0/1105359487 shutdown_connections 2024-03-16T20:08:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.003+0000 7feeae736640 1 -- 172.21.15.32:0/1105359487 wait complete. 2024-03-16T20:08:50.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 Processor -- start 2024-03-16T20:08:50.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 -- start start 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6420 0x7feea81560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feea8158390 0x7feea8148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea81566d0 con 0x7feea80c6040 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea81573e0 con 0x7feea80c6420 2024-03-16T20:08:50.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeae736640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea8148220 con 0x7feea8158390 2024-03-16T20:08:50.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feea7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:50.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feea77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6420 0x7feea81560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:50.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feeaccd3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feea8158390 0x7feea8148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feea7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44420/0 (socket says 172.21.15.32:44420) 2024-03-16T20:08:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.007+0000 7feea7fff640 1 -- 172.21.15.32:0/1631912386 learned_addr learned my addr 172.21.15.32:0/1631912386 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 -- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feea8158390 msgr2=0x7feea8148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feea8158390 0x7feea8148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 -- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6420 msgr2=0x7feea81560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6420 0x7feea81560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 -- 172.21.15.32:0/1631912386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980094d0 con 0x7feea80c6040 2024-03-16T20:08:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.015+0000 7feea7fff640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee9801a0e0 tx=0x7fee9801acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980b3800 con 0x7feea80c6040 2024-03-16T20:08:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee980b3980 con 0x7feea80c6040 2024-03-16T20:08:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea81491b0 con 0x7feea80c6040 2024-03-16T20:08:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980b3d30 con 0x7feea80c6040 2024-03-16T20:08:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea8157a20 con 0x7feea80c6040 2024-03-16T20:08:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.023+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea80729b0 con 0x7feea80c6040 2024-03-16T20:08:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.059+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fee980c7420 con 0x7feea80c6040 2024-03-16T20:08:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.059+0000 7feea4ff9640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 0x7fee94069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.059+0000 7feea77fe640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 0x7fee94069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:51.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.059+0000 7feea77fe640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 0x7fee94069350 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feea806c770 tx=0x7fee9c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:51.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.063+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fee980aae20 con 0x7feea80c6040 2024-03-16T20:08:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.363+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fee980ec210 con 0x7feea80c6040 2024-03-16T20:08:51.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.659+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea80729b0 con 0x7feea80c6040 2024-03-16T20:08:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.679+0000 7feea4ff9640 1 -- 172.21.15.32:0/1631912386 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fee980ef8d0 con 0x7feea80c6040 2024-03-16T20:08:51.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:51.671 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 msgr2=0x7fee94069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 0x7fee94069350 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feea806c770 tx=0x7fee9c007e40 comp rx=0 tx=0).stop 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 msgr2=0x7feea814cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee9801a0e0 tx=0x7fee9801acf0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 shutdown_connections 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fee94066e90 0x7fee94069350 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feea8158390 0x7feea8148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feea80c6420 0x7feea81560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 --2- 172.21.15.32:0/1631912386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea80c6040 0x7feea814cd20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 >> 172.21.15.32:0/1631912386 conn(0x7feea806c0d0 msgr2=0x7feea80cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 shutdown_connections 2024-03-16T20:08:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:51.683+0000 7feeae736640 1 -- 172.21.15.32:0/1631912386 wait complete. 2024-03-16T20:08:52.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:52.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:53.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.411+0000 7fc83e5f5640 1 Processor -- start 2024-03-16T20:08:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.411+0000 7fc83e5f5640 1 -- start start 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83e5f5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc838154130 0x7fc838070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83e5f5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc838070970 0x7fc838072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83e5f5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83815ae90 con 0x7fc8380722d0 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83e5f5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc838159ec0 con 0x7fc838070970 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83e5f5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc838072830 con 0x7fc838154130 2024-03-16T20:08:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83cb92640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc838070970 0x7fc838072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc837fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc838154130 0x7fc838070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83cb92640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc838070970 0x7fc838072b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44440/0 (socket says 172.21.15.32:44440) 2024-03-16T20:08:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.415+0000 7fc83cb92640 1 -- 172.21.15.32:0/3717850276 learned_addr learned my addr 172.21.15.32:0/3717850276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.419+0000 7fc83cb92640 1 -- 172.21.15.32:0/3717850276 >> v2:172.21.15.59:3300/0 conn(0x7fc838154130 msgr2=0x7fc838070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.419+0000 7fc83cb92640 1 --2- 172.21.15.32:0/3717850276 >> v2:172.21.15.59:3300/0 conn(0x7fc838154130 0x7fc838070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.419+0000 7fc83cb92640 1 -- 172.21.15.32:0/3717850276 >> v1:172.21.15.59:6789/0 conn(0x7fc8380722d0 legacy=0x7fc83815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.419+0000 7fc83cb92640 1 -- 172.21.15.32:0/3717850276 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83815a0d0 con 0x7fc838070970 2024-03-16T20:08:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.423+0000 7fc83cb92640 1 --2- 172.21.15.32:0/3717850276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc838072b10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc838070940 tx=0x7fc83001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec0c814ea1ef30b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc836ffd640 1 -- 172.21.15.32:0/3717850276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300af400 con 0x7fc838070970 2024-03-16T20:08:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc836ffd640 1 -- 172.21.15.32:0/3717850276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8300ab050 con 0x7fc838070970 2024-03-16T20:08:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc836ffd640 1 -- 172.21.15.32:0/3717850276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300af780 con 0x7fc838070970 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 -- 172.21.15.32:0/3717850276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 msgr2=0x7fc838072b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/3717850276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc838072b10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc838070940 tx=0x7fc83001f1e0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 -- 172.21.15.32:0/3717850276 shutdown_connections 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/3717850276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc838072b10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/3717850276 >> v2:172.21.15.59:3300/0 conn(0x7fc838154130 0x7fc838070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 -- 172.21.15.32:0/3717850276 >> 172.21.15.32:0/3717850276 conn(0x7fc83806c0d0 msgr2=0x7fc83806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 -- 172.21.15.32:0/3717850276 shutdown_connections 2024-03-16T20:08:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.427+0000 7fc83e5f5640 1 -- 172.21.15.32:0/3717850276 wait complete. 2024-03-16T20:08:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 Processor -- start 2024-03-16T20:08:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 -- start start 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380722d0 0x7fc8380cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc838154130 0x7fc8380cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838152340 con 0x7fc838070970 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8381527d0 con 0x7fc8380722d0 2024-03-16T20:08:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83e5f5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8381f0e70 con 0x7fc838154130 2024-03-16T20:08:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc837fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc8377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380722d0 0x7fc8380cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc837fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44446/0 (socket says 172.21.15.32:44446) 2024-03-16T20:08:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc837fff640 1 -- 172.21.15.32:0/1245944362 learned_addr learned my addr 172.21.15.32:0/1245944362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.431+0000 7fc83cb92640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc838154130 0x7fc8380cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.439+0000 7fc837fff640 1 -- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc838154130 msgr2=0x7fc8380cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.439+0000 7fc837fff640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc838154130 0x7fc8380cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.439+0000 7fc837fff640 1 -- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380722d0 msgr2=0x7fc8380cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.439+0000 7fc837fff640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380722d0 0x7fc8380cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.439+0000 7fc837fff640 1 -- 172.21.15.32:0/1245944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc830009b30 con 0x7fc838070970 2024-03-16T20:08:53.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.443+0000 7fc837fff640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc82800d830 tx=0x7fc82800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82800b6c0 con 0x7fc838070970 2024-03-16T20:08:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82800b820 con 0x7fc838070970 2024-03-16T20:08:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838151730 con 0x7fc838070970 2024-03-16T20:08:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82800bb90 con 0x7fc838070970 2024-03-16T20:08:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc838151cf0 con 0x7fc838070970 2024-03-16T20:08:53.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.447+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838068000 con 0x7fc838070970 2024-03-16T20:08:53.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.483+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc82801e460 con 0x7fc838070970 2024-03-16T20:08:53.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.483+0000 7fc834ff9640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 0x7fc824069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:53.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.483+0000 7fc8377fe640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 0x7fc824069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:53.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.487+0000 7fc8377fe640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 0x7fc824069350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc8380cd740 tx=0x7fc82c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:53.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.495+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc828056760 con 0x7fc838070970 2024-03-16T20:08:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:53.839+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc828052a00 con 0x7fc838070970 2024-03-16T20:08:54.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.047+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8380c6990 con 0x7fc838070970 2024-03-16T20:08:54.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc834ff9640 1 -- 172.21.15.32:0/1245944362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc8280560c0 con 0x7fc838070970 2024-03-16T20:08:54.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:54.061 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:54.061 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 msgr2=0x7fc824069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 0x7fc824069350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc8380cd740 tx=0x7fc82c009040 comp rx=0 tx=0).stop 2024-03-16T20:08:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 msgr2=0x7fc83815c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc82800d830 tx=0x7fc82800dcf0 comp rx=0 tx=0).stop 2024-03-16T20:08:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 shutdown_connections 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc824066e90 0x7fc824069350 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc838154130 0x7fc8380cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380722d0 0x7fc8380cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 --2- 172.21.15.32:0/1245944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc838070970 0x7fc83815c6b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 >> 172.21.15.32:0/1245944362 conn(0x7fc83806c0d0 msgr2=0x7fc8380c7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 shutdown_connections 2024-03-16T20:08:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:54.071+0000 7fc83e5f5640 1 -- 172.21.15.32:0/1245944362 wait complete. 2024-03-16T20:08:54.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:54.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:55.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 Processor -- start 2024-03-16T20:08:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 -- start start 2024-03-16T20:08:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39c40c6040 0x7f39c40c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c4070760 con 0x7f39c40c6420 2024-03-16T20:08:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40731c0 con 0x7f39c4158390 2024-03-16T20:08:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39ca35b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40734a0 con 0x7f39c40c6040 2024-03-16T20:08:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39c3fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39c40c6040 0x7f39c40c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39c88f8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39c4158390 0x7f39c41516d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:46220/0 (socket says 172.21.15.32:46220) 2024-03-16T20:08:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.807+0000 7f39c88f8640 1 -- 172.21.15.32:0/1901095507 learned_addr learned my addr 172.21.15.32:0/1901095507 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:08:55.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.811+0000 7f39c3fff640 1 -- 172.21.15.32:0/1901095507 >> v1:172.21.15.59:6789/0 conn(0x7f39c4158390 legacy=0x7f39c41516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:55.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.811+0000 7f39c3fff640 1 -- 172.21.15.32:0/1901095507 >> v1:172.21.15.32:6789/0 conn(0x7f39c40c6420 legacy=0x7f39c4157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:55.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.811+0000 7f39c3fff640 1 -- 172.21.15.32:0/1901095507 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c406fe30 con 0x7f39c40c6040 2024-03-16T20:08:55.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.819+0000 7f39c3fff640 1 --2- 172.21.15.32:0/1901095507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6040 0x7f39c40c8800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f39b40097c0 tx=0x7f39b401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951a1cfaf663360a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:55.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39c2ffd640 1 -- 172.21.15.32:0/1901095507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b40ab670 con 0x7f39c40c6040 2024-03-16T20:08:55.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39c2ffd640 1 -- 172.21.15.32:0/1901095507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b40ab7d0 con 0x7f39c40c6040 2024-03-16T20:08:55.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39c2ffd640 1 -- 172.21.15.32:0/1901095507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b40abb60 con 0x7f39c40c6040 2024-03-16T20:08:55.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 -- 172.21.15.32:0/1901095507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6040 msgr2=0x7f39c40c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 --2- 172.21.15.32:0/1901095507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6040 0x7f39c40c8800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f39b40097c0 tx=0x7f39b401c470 comp rx=0 tx=0).stop 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 -- 172.21.15.32:0/1901095507 shutdown_connections 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 --2- 172.21.15.32:0/1901095507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6040 0x7f39c40c8800 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 -- 172.21.15.32:0/1901095507 >> 172.21.15.32:0/1901095507 conn(0x7f39c406c0d0 msgr2=0x7f39c406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 -- 172.21.15.32:0/1901095507 shutdown_connections 2024-03-16T20:08:55.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.823+0000 7f39ca35b640 1 -- 172.21.15.32:0/1901095507 wait complete. 2024-03-16T20:08:55.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 Processor -- start 2024-03-16T20:08:55.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 -- start start 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c40c6040 0x7f39c40c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c4158390 0x7f39c4141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c414f5e0 con 0x7f39c40c6420 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c4153610 con 0x7f39c40c6040 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39ca35b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c4148060 con 0x7f39c4158390 2024-03-16T20:08:55.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39c37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.827+0000 7f39c88f8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c4158390 0x7f39c4141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.831+0000 7f39c37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44468/0 (socket says 172.21.15.32:44468) 2024-03-16T20:08:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.831+0000 7f39c37fe640 1 -- 172.21.15.32:0/2133320660 learned_addr learned my addr 172.21.15.32:0/2133320660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.831+0000 7f39c3fff640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c40c6040 0x7f39c40c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.835+0000 7f39c37fe640 1 -- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c4158390 msgr2=0x7f39c4141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.835+0000 7f39c37fe640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c4158390 0x7f39c4141c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.835+0000 7f39c37fe640 1 -- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c40c6040 msgr2=0x7f39c40c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.835+0000 7f39c37fe640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c40c6040 0x7f39c40c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.835+0000 7f39c37fe640 1 -- 172.21.15.32:0/2133320660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4009420 con 0x7f39c40c6420 2024-03-16T20:08:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.839+0000 7f39c37fe640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f39b800c430 tx=0x7f39b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b80052d0 con 0x7f39c40c6420 2024-03-16T20:08:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b8005430 con 0x7f39c40c6420 2024-03-16T20:08:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8007260 con 0x7f39c40c6420 2024-03-16T20:08:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c41f4d00 con 0x7f39c40c6420 2024-03-16T20:08:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c4153990 con 0x7f39c40c6420 2024-03-16T20:08:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:55.843+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c4068000 con 0x7f39c40c6420 2024-03-16T20:08:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.123+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f39b80073c0 con 0x7f39c40c6420 2024-03-16T20:08:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.123+0000 7f39c0ff9640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 0x7f39b0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.123+0000 7f39c3fff640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 0x7f39b0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.123+0000 7f39c3fff640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 0x7f39b0069330 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f39b40af000 tx=0x7f39b40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:56.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.159+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f39b8007670 con 0x7f39c40c6420 2024-03-16T20:08:56.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.159+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f39b8051420 con 0x7f39c40c6420 2024-03-16T20:08:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.387+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c40c3280 con 0x7f39c40c6420 2024-03-16T20:08:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.411+0000 7f39c0ff9640 1 -- 172.21.15.32:0/2133320660 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f39b8020020 con 0x7f39c40c6420 2024-03-16T20:08:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:56.403 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:56.403 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 msgr2=0x7f39b0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 0x7f39b0069330 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f39b40af000 tx=0x7f39b40a7000 comp rx=0 tx=0).stop 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 msgr2=0x7f39c41416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f39b800c430 tx=0x7f39b800c8f0 comp rx=0 tx=0).stop 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 shutdown_connections 2024-03-16T20:08:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f39b0066e70 0x7f39b0069330 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c4158390 0x7f39c4141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39c40c6420 0x7f39c41416d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 --2- 172.21.15.32:0/2133320660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c40c6040 0x7f39c40c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 >> 172.21.15.32:0/2133320660 conn(0x7f39c406c0d0 msgr2=0x7f39c414ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 shutdown_connections 2024-03-16T20:08:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:56.415+0000 7f39ca35b640 1 -- 172.21.15.32:0/2133320660 wait complete. 2024-03-16T20:08:57.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:57.031 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:08:58.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:08:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 Processor -- start 2024-03-16T20:08:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 -- start start 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f084c154350 0x7f084c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c072a10 0x7f084c072e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c06bae0 con 0x7f084c0c2d70 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c15a540 con 0x7f084c154350 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0852790640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0c7fd0 con 0x7f084c072a10 2024-03-16T20:08:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f084bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f084c154350 0x7f084c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f084bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f084c154350 0x7f084c06c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44484/0 (socket says 172.21.15.32:44484) 2024-03-16T20:08:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f084b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f084c0c2d70 0x7f084c15a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:46234/0 (socket says 172.21.15.32:46234) 2024-03-16T20:08:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f084bfff640 1 -- 172.21.15.32:0/25397801 learned_addr learned my addr 172.21.15.32:0/25397801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.151+0000 7f0850d2d640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c072a10 0x7f084c072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.155+0000 7f084bfff640 1 -- 172.21.15.32:0/25397801 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c072a10 msgr2=0x7f084c072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:08:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.155+0000 7f084bfff640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c072a10 0x7f084c072e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.155+0000 7f084bfff640 1 -- 172.21.15.32:0/25397801 >> v1:172.21.15.59:6789/0 conn(0x7f084c0c2d70 legacy=0x7f084c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.155+0000 7f084bfff640 1 -- 172.21.15.32:0/25397801 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c06bd10 con 0x7f084c154350 2024-03-16T20:08:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.159+0000 7f084bfff640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c154350 0x7f084c06c250 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68c0700ec1251ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.159+0000 7f084affd640 1 -- 172.21.15.32:0/25397801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c0ac620 con 0x7f084c154350 2024-03-16T20:08:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f084affd640 1 -- 172.21.15.32:0/25397801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c0ac780 con 0x7f084c154350 2024-03-16T20:08:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f084affd640 1 -- 172.21.15.32:0/25397801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c0acb10 con 0x7f084c154350 2024-03-16T20:08:58.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 -- 172.21.15.32:0/25397801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c154350 msgr2=0x7f084c06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c154350 0x7f084c06c250 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c01c4d0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 -- 172.21.15.32:0/25397801 shutdown_connections 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c072a10 0x7f084c072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 --2- 172.21.15.32:0/25397801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c154350 0x7f084c06c250 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 -- 172.21.15.32:0/25397801 >> 172.21.15.32:0/25397801 conn(0x7f084c06afc0 msgr2=0x7f084c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 -- 172.21.15.32:0/25397801 shutdown_connections 2024-03-16T20:08:58.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.163+0000 7f0852790640 1 -- 172.21.15.32:0/25397801 wait complete. 2024-03-16T20:08:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 Processor -- start 2024-03-16T20:08:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 -- start start 2024-03-16T20:08:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f084c072a10 0x7f084c14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c0c2d70 0x7f084c14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1518d0 con 0x7f084c15b220 2024-03-16T20:08:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c14c170 con 0x7f084c072a10 2024-03-16T20:08:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0852790640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15ba00 con 0x7f084c0c2d70 2024-03-16T20:08:58.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0850d2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f084bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f084c072a10 0x7f084c14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0850d2d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44496/0 (socket says 172.21.15.32:44496) 2024-03-16T20:08:58.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f0850d2d640 1 -- 172.21.15.32:0/1829752747 learned_addr learned my addr 172.21.15.32:0/1829752747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:08:58.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.167+0000 7f084b7fe640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c0c2d70 0x7f084c14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 -- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c0c2d70 msgr2=0x7f084c14e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c0c2d70 0x7f084c14e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 -- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f084c072a10 msgr2=0x7f084c14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f084c072a10 0x7f084c14bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 -- 172.21.15.32:0/1829752747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0094d0 con 0x7f084c15b220 2024-03-16T20:08:58.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.175+0000 7f0850d2d640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0844009dd0 tx=0x7f084400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:58.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844014480 con 0x7f084c15b220 2024-03-16T20:08:58.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400f040 con 0x7f084c15b220 2024-03-16T20:08:58.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844014840 con 0x7f084c15b220 2024-03-16T20:08:58.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c15bca0 con 0x7f084c15b220 2024-03-16T20:08:58.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c151210 con 0x7f084c15b220 2024-03-16T20:08:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.183+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c0674e0 con 0x7f084c15b220 2024-03-16T20:08:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.203+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08440149a0 con 0x7f084c15b220 2024-03-16T20:08:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.203+0000 7f0848ff9640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 0x7f0838069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:08:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.207+0000 7f084bfff640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 0x7f0838069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:08:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.207+0000 7f084bfff640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 0x7f0838069350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f083c0ae000 tx=0x7f083c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.207+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0844056960 con 0x7f084c15b220 2024-03-16T20:08:58.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.479+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0844052c00 con 0x7f084c15b220 2024-03-16T20:08:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.675+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f084c068000 con 0x7f084c15b220 2024-03-16T20:08:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.695+0000 7f0848ff9640 1 -- 172.21.15.32:0/1829752747 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f0844018090 con 0x7f084c15b220 2024-03-16T20:08:58.687 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:08:58.687 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:08:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 msgr2=0x7f0838069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 0x7f0838069350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f083c0ae000 tx=0x7f083c0a7000 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 msgr2=0x7f084c14ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0844009dd0 tx=0x7f084400c490 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 shutdown_connections 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0838066e90 0x7f0838069350 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f084c15b220 0x7f084c14ea60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f084c0c2d70 0x7f084c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 --2- 172.21.15.32:0/1829752747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f084c072a10 0x7f084c14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 >> 172.21.15.32:0/1829752747 conn(0x7f084c06afc0 msgr2=0x7f084c158de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 shutdown_connections 2024-03-16T20:08:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:08:58.699+0000 7f0852790640 1 -- 172.21.15.32:0/1829752747 wait complete. 2024-03-16T20:08:59.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:08:59.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:00.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 Processor -- start 2024-03-16T20:09:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 -- start start 2024-03-16T20:09:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 0x7fc2641658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2641546a0 con 0x7fc2641542c0 2024-03-16T20:09:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc264159f20 con 0x7fc264070360 2024-03-16T20:09:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26bf4f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415a1e0 con 0x7fc264073010 2024-03-16T20:09:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc269ceb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc2641542c0 0x7fc264072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58948/0 (socket says 172.21.15.32:58948) 2024-03-16T20:09:00.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc269ceb640 1 -- 172.21.15.32:0/3502663022 learned_addr learned my addr 172.21.15.32:0/3502663022 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:00.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.427+0000 7fc26a4ec640 1 --2- 172.21.15.32:0/3502663022 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 0x7fc2641658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.431+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842967127 0 0) 0x7fc2641546a0 con 0x7fc2641542c0 2024-03-16T20:09:00.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.431+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc250003530 con 0x7fc2641542c0 2024-03-16T20:09:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.435+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814517854 0 0) 0x7fc264159f20 con 0x7fc264070360 2024-03-16T20:09:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.435+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2641546a0 con 0x7fc264070360 2024-03-16T20:09:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.435+0000 7fc26a4ec640 1 -- 172.21.15.32:0/3502663022 >> v1:172.21.15.59:6789/0 conn(0x7fc264070360 legacy=0x7fc26415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.435+0000 7fc26a4ec640 1 -- 172.21.15.32:0/3502663022 >> v1:172.21.15.32:6789/0 conn(0x7fc2641542c0 legacy=0x7fc264072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.435+0000 7fc26a4ec640 1 -- 172.21.15.32:0/3502663022 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26415a4c0 con 0x7fc264073010 2024-03-16T20:09:00.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.443+0000 7fc26a4ec640 1 --2- 172.21.15.32:0/3502663022 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 0x7fc2641658b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009dc0 tx=0x7fc25c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9526c767f5424099 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0a5940 con 0x7fc264073010 2024-03-16T20:09:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c0a5aa0 con 0x7fc264073010 2024-03-16T20:09:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc268ce9640 1 -- 172.21.15.32:0/3502663022 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0a5e30 con 0x7fc264073010 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- 172.21.15.32:0/3502663022 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 msgr2=0x7fc2641658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/3502663022 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 0x7fc2641658b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009dc0 tx=0x7fc25c01c990 comp rx=0 tx=0).stop 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- 172.21.15.32:0/3502663022 shutdown_connections 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/3502663022 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264073010 0x7fc2641658b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- 172.21.15.32:0/3502663022 >> 172.21.15.32:0/3502663022 conn(0x7fc26406c0d0 msgr2=0x7fc26406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- 172.21.15.32:0/3502663022 shutdown_connections 2024-03-16T20:09:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- 172.21.15.32:0/3502663022 wait complete. 2024-03-16T20:09:00.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 Processor -- start 2024-03-16T20:09:00.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- start start 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264070360 0x7fc26415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 0x7fc26414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 0x7fc2641f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415d430 con 0x7fc2641542c0 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2641f3a10 con 0x7fc264073010 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26bf4f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc264149c30 con 0x7fc264070360 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc2694ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 0x7fc26414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.451+0000 7fc26a4ec640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 0x7fc2641f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.455+0000 7fc2694ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 0x7fc26414a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:52114/0 (socket says 172.21.15.32:52114) 2024-03-16T20:09:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.455+0000 7fc2694ea640 1 -- 172.21.15.32:0/502717731 learned_addr learned my addr 172.21.15.32:0/502717731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.455+0000 7fc269ceb640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264070360 0x7fc26415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.459+0000 7fc26a4ec640 1 -- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264070360 msgr2=0x7fc26415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.459+0000 7fc26a4ec640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264070360 0x7fc26415b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.459+0000 7fc26a4ec640 1 -- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 msgr2=0x7fc26414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.459+0000 7fc26a4ec640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 0x7fc26414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.459+0000 7fc26a4ec640 1 -- 172.21.15.32:0/502717731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c009a70 con 0x7fc2641542c0 2024-03-16T20:09:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.463+0000 7fc26a4ec640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 0x7fc2641f34d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0ac000 tx=0x7fc25c019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0aebf0 con 0x7fc2641542c0 2024-03-16T20:09:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c0aed50 con 0x7fc2641542c0 2024-03-16T20:09:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0b7650 con 0x7fc2641542c0 2024-03-16T20:09:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26415ce70 con 0x7fc2641542c0 2024-03-16T20:09:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc264149ed0 con 0x7fc2641542c0 2024-03-16T20:09:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.467+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc264149a90 con 0x7fc2641542c0 2024-03-16T20:09:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.503+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc25c0b4050 con 0x7fc2641542c0 2024-03-16T20:09:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.503+0000 7fc25a7fc640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 0x7fc248069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.503+0000 7fc269ceb640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 0x7fc248069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:00.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.503+0000 7fc269ceb640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 0x7fc248069380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc260007fd0 tx=0x7fc260000bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:00.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.507+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc25c0f0de0 con 0x7fc2641542c0 2024-03-16T20:09:00.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:00.803+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc25c0ed080 con 0x7fc2641542c0 2024-03-16T20:09:01.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.019+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc264149a90 con 0x7fc2641542c0 2024-03-16T20:09:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc25a7fc640 1 -- 172.21.15.32:0/502717731 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc264149a90 con 0x7fc2641542c0 2024-03-16T20:09:01.033 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:01.033 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 msgr2=0x7fc248069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 0x7fc248069380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc260007fd0 tx=0x7fc260000bd0 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 msgr2=0x7fc2641f34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 0x7fc2641f34d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0ac000 tx=0x7fc25c019730 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 shutdown_connections 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc248066ec0 0x7fc248069380 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2641542c0 0x7fc2641f34d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc264073010 0x7fc26414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 --2- 172.21.15.32:0/502717731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc264070360 0x7fc26415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 >> 172.21.15.32:0/502717731 conn(0x7fc26406c0d0 msgr2=0x7fc26406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 shutdown_connections 2024-03-16T20:09:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:01.043+0000 7fc26bf4f640 1 -- 172.21.15.32:0/502717731 wait complete. 2024-03-16T20:09:01.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:01.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:02.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 Processor -- start 2024-03-16T20:09:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 -- start start 2024-03-16T20:09:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa11c0725e0 0x7fa11c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c072dd0 con 0x7fa11c150d70 2024-03-16T20:09:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c15d790 con 0x7fa11c15a420 2024-03-16T20:09:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.775+0000 7fa121755640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c15da50 con 0x7fa11c0725e0 2024-03-16T20:09:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa11c0725e0 0x7fa11c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa11c0725e0 0x7fa11c073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33072/0 (socket says 172.21.15.32:33072) 2024-03-16T20:09:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 -- 172.21.15.32:0/3391453112 learned_addr learned my addr 172.21.15.32:0/3391453112 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 -- 172.21.15.32:0/3391453112 >> v1:172.21.15.59:6789/0 conn(0x7fa11c15a420 legacy=0x7fa11c15a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 -- 172.21.15.32:0/3391453112 >> v1:172.21.15.32:6789/0 conn(0x7fa11c150d70 legacy=0x7fa11c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.779+0000 7fa11a7fc640 1 -- 172.21.15.32:0/3391453112 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c15dd30 con 0x7fa11c0725e0 2024-03-16T20:09:02.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.787+0000 7fa11a7fc640 1 --2- 172.21.15.32:0/3391453112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c0725e0 0x7fa11c073070 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa110009870 tx=0x7fa11001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370993974f4bec0c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa119ffb640 1 -- 172.21.15.32:0/3391453112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1100ac630 con 0x7fa11c0725e0 2024-03-16T20:09:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa119ffb640 1 -- 172.21.15.32:0/3391453112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1100ac790 con 0x7fa11c0725e0 2024-03-16T20:09:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa119ffb640 1 -- 172.21.15.32:0/3391453112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1100acb20 con 0x7fa11c0725e0 2024-03-16T20:09:02.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- 172.21.15.32:0/3391453112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c0725e0 msgr2=0x7fa11c073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:02.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 --2- 172.21.15.32:0/3391453112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c0725e0 0x7fa11c073070 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa110009870 tx=0x7fa11001c3d0 comp rx=0 tx=0).stop 2024-03-16T20:09:02.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- 172.21.15.32:0/3391453112 shutdown_connections 2024-03-16T20:09:02.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 --2- 172.21.15.32:0/3391453112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c0725e0 0x7fa11c073070 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:02.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- 172.21.15.32:0/3391453112 >> 172.21.15.32:0/3391453112 conn(0x7fa11c06c0d0 msgr2=0x7fa11c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:02.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- 172.21.15.32:0/3391453112 shutdown_connections 2024-03-16T20:09:02.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- 172.21.15.32:0/3391453112 wait complete. 2024-03-16T20:09:02.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 Processor -- start 2024-03-16T20:09:02.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.791+0000 7fa121755640 1 -- start start 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0725e0 0x7fa11c146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa11c150d70 0x7fa11c1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c1463c0 con 0x7fa11c15a420 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c1f60d0 con 0x7fa11c0725e0 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa121755640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c1457f0 con 0x7fa11c150d70 2024-03-16T20:09:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa11b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa11a7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa11c150d70 0x7fa11c1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa11affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0725e0 0x7fa11c146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa11b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33088/0 (socket says 172.21.15.32:33088) 2024-03-16T20:09:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.795+0000 7fa11b7fe640 1 -- 172.21.15.32:0/66585386 learned_addr learned my addr 172.21.15.32:0/66585386 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:02.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.799+0000 7fa11b7fe640 1 -- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa11c150d70 msgr2=0x7fa11c1511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.799+0000 7fa11b7fe640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa11c150d70 0x7fa11c1511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.799+0000 7fa11b7fe640 1 -- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0725e0 msgr2=0x7fa11c146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.799+0000 7fa11b7fe640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0725e0 0x7fa11c146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.799+0000 7fa11b7fe640 1 -- 172.21.15.32:0/66585386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100094d0 con 0x7fa11c15a420 2024-03-16T20:09:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.823+0000 7fa11b7fe640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa11400c710 tx=0x7fa11400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.827+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114006d60 con 0x7fa11c15a420 2024-03-16T20:09:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.827+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c15d560 con 0x7fa11c15a420 2024-03-16T20:09:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.827+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c145d70 con 0x7fa11c15a420 2024-03-16T20:09:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:02.827+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c146530 con 0x7fa11c15a420 2024-03-16T20:09:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.127+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11400ce20 con 0x7fa11c15a420 2024-03-16T20:09:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.127+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114003ba0 con 0x7fa11c15a420 2024-03-16T20:09:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.131+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa11400a850 con 0x7fa11c15a420 2024-03-16T20:09:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.131+0000 7fa1037fe640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 0x7fa108069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.131+0000 7fa11affd640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 0x7fa108069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.131+0000 7fa11affd640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 0x7fa108069300 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0098e0 tx=0x7fa10c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.155+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fa114055f30 con 0x7fa11c15a420 2024-03-16T20:09:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.155+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fa1140521d0 con 0x7fa11c15a420 2024-03-16T20:09:03.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.363+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa11c068000 con 0x7fa11c15a420 2024-03-16T20:09:03.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa1037fe640 1 -- 172.21.15.32:0/66585386 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fa114055890 con 0x7fa11c15a420 2024-03-16T20:09:03.421 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:03.421 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:03.422 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 msgr2=0x7fa108069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 0x7fa108069300 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0098e0 tx=0x7fa10c009060 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 msgr2=0x7fa11c0cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa11400c710 tx=0x7fa11400cbd0 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 shutdown_connections 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fa108066e40 0x7fa108069300 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa11c15a420 0x7fa11c0cc910 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa11c150d70 0x7fa11c1511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 --2- 172.21.15.32:0/66585386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0725e0 0x7fa11c146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 >> 172.21.15.32:0/66585386 conn(0x7fa11c06c0d0 msgr2=0x7fa11c0cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.431+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 shutdown_connections 2024-03-16T20:09:03.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:03.435+0000 7fa121755640 1 -- 172.21.15.32:0/66585386 wait complete. 2024-03-16T20:09:04.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:04.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:05.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 Processor -- start 2024-03-16T20:09:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 -- start start 2024-03-16T20:09:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b80c36b0 0x7f53b80c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53b80c8980 0x7f53b815a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80c8ec0 con 0x7f53b8151000 2024-03-16T20:09:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b8072f10 con 0x7f53b80c8980 2024-03-16T20:09:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.163+0000 7f53be4e7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80c5000 con 0x7f53b80c36b0 2024-03-16T20:09:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.167+0000 7f53bca84640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53b80c8980 0x7f53b815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.167+0000 7f53bca84640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53b80c8980 0x7f53b815a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33096/0 (socket says 172.21.15.32:33096) 2024-03-16T20:09:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.167+0000 7f53bca84640 1 -- 172.21.15.32:0/2919901842 learned_addr learned my addr 172.21.15.32:0/2919901842 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.167+0000 7f53b77fe640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b80c36b0 0x7f53b80c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.171+0000 7f53bca84640 1 -- 172.21.15.32:0/2919901842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b80c36b0 msgr2=0x7f53b80c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.171+0000 7f53bca84640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b80c36b0 0x7f53b80c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.171+0000 7f53bca84640 1 -- 172.21.15.32:0/2919901842 >> v1:172.21.15.59:6789/0 conn(0x7f53b8151000 legacy=0x7f53b80c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.171+0000 7f53bca84640 1 -- 172.21.15.32:0/2919901842 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b8072950 con 0x7f53b80c8980 2024-03-16T20:09:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.175+0000 7f53bca84640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b815a240 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009e50 tx=0x7f53b001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239e792ce3d17432 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.175+0000 7f53b6ffd640 1 -- 172.21.15.32:0/2919901842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b00a5900 con 0x7f53b80c8980 2024-03-16T20:09:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53b6ffd640 1 -- 172.21.15.32:0/2919901842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b00a5a60 con 0x7f53b80c8980 2024-03-16T20:09:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53b6ffd640 1 -- 172.21.15.32:0/2919901842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b00a5df0 con 0x7f53b80c8980 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 -- 172.21.15.32:0/2919901842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 msgr2=0x7f53b815a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b815a240 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009e50 tx=0x7f53b001ca30 comp rx=0 tx=0).stop 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 -- 172.21.15.32:0/2919901842 shutdown_connections 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b815a240 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 --2- 172.21.15.32:0/2919901842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b80c36b0 0x7f53b80c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 -- 172.21.15.32:0/2919901842 >> 172.21.15.32:0/2919901842 conn(0x7f53b806c0d0 msgr2=0x7f53b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 -- 172.21.15.32:0/2919901842 shutdown_connections 2024-03-16T20:09:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.179+0000 7f53be4e7640 1 -- 172.21.15.32:0/2919901842 wait complete. 2024-03-16T20:09:05.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 Processor -- start 2024-03-16T20:09:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 -- start start 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53b80c36b0 0x7f53b8146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b8151000 0x7f53b8142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b81585a0 con 0x7f53b80c8980 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8157e30 con 0x7f53b80c36b0 2024-03-16T20:09:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53be4e7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8159e50 con 0x7f53b8151000 2024-03-16T20:09:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53b80c36b0 0x7f53b8146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33108/0 (socket says 172.21.15.32:33108) 2024-03-16T20:09:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53b77fe640 1 -- 172.21.15.32:0/317126562 learned_addr learned my addr 172.21.15.32:0/317126562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.183+0000 7f53bca84640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b8151000 0x7f53b8142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.191+0000 7f53b77fe640 1 -- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b8151000 msgr2=0x7f53b8142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.191+0000 7f53b77fe640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b8151000 0x7f53b8142460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.191+0000 7f53b77fe640 1 -- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53b80c36b0 msgr2=0x7f53b8146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.191+0000 7f53b77fe640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53b80c36b0 0x7f53b8146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.191+0000 7f53b77fe640 1 -- 172.21.15.32:0/317126562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b0009b30 con 0x7f53b80c8980 2024-03-16T20:09:05.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.195+0000 7f53b77fe640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00e960 tx=0x7f53ac00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac009690 con 0x7f53b80c8980 2024-03-16T20:09:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac00ca00 con 0x7f53b80c8980 2024-03-16T20:09:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00cd70 con 0x7f53b80c8980 2024-03-16T20:09:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b8160190 con 0x7f53b80c8980 2024-03-16T20:09:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b8158080 con 0x7f53b80c8980 2024-03-16T20:09:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.199+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b8068000 con 0x7f53b80c8980 2024-03-16T20:09:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.223+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f53ac013470 con 0x7f53b80c8980 2024-03-16T20:09:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.223+0000 7f53b4ff9640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 0x7f53a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.223+0000 7f53b7fff640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 0x7f53a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.223+0000 7f53b7fff640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 0x7f53a4069350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007fd0 tx=0x7f53a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.507+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f53ac055f80 con 0x7f53b80c8980 2024-03-16T20:09:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.507+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f53ac052220 con 0x7f53b80c8980 2024-03-16T20:09:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.707+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53b8073410 con 0x7f53b80c8980 2024-03-16T20:09:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53b4ff9640 1 -- 172.21.15.32:0/317126562 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f53ac021090 con 0x7f53b80c8980 2024-03-16T20:09:05.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:05.781 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 msgr2=0x7f53a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 0x7f53a4069350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007fd0 tx=0x7f53a8009040 comp rx=0 tx=0).stop 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 msgr2=0x7f53b8141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00e960 tx=0x7f53ac00ee20 comp rx=0 tx=0).stop 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 shutdown_connections 2024-03-16T20:09:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f53a4066e90 0x7f53a4069350 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f53b8151000 0x7f53b8142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53b80c8980 0x7f53b8141f20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 --2- 172.21.15.32:0/317126562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53b80c36b0 0x7f53b8146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.791+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 >> 172.21.15.32:0/317126562 conn(0x7f53b806c0d0 msgr2=0x7f53b814e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.795+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 shutdown_connections 2024-03-16T20:09:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:05.795+0000 7f53be4e7640 1 -- 172.21.15.32:0/317126562 wait complete. 2024-03-16T20:09:06.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:06.410 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:07.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:07.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 Processor -- start 2024-03-16T20:09:07.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 -- start start 2024-03-16T20:09:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f48dc15a420 0x7f48dc15a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc072dd0 con 0x7f48dc0725e0 2024-03-16T20:09:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc15d790 con 0x7f48dc15a420 2024-03-16T20:09:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.539+0000 7f48e1a46640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc15ad90 con 0x7f48dc150d70 2024-03-16T20:09:07.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48da7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f48dc0725e0 0x7f48dc070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58974/0 (socket says 172.21.15.32:58974) 2024-03-16T20:09:07.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48da7fc640 1 -- 172.21.15.32:0/2413946111 learned_addr learned my addr 172.21.15.32:0/2413946111 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:07.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48daffd640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48db7fe640 1 --2- 172.21.15.32:0/2413946111 >> v2:172.21.15.59:3300/0 conn(0x7f48dc15a420 0x7f48dc15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48d9ffb640 1 -- 172.21.15.32:0/2413946111 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621154334 0 0) 0x7f48dc072dd0 con 0x7f48dc0725e0 2024-03-16T20:09:07.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.543+0000 7f48d9ffb640 1 -- 172.21.15.32:0/2413946111 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c4003530 con 0x7f48dc0725e0 2024-03-16T20:09:07.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.547+0000 7f48db7fe640 1 -- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 msgr2=0x7f48dc073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:07.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.547+0000 7f48db7fe640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.547+0000 7f48db7fe640 1 -- 172.21.15.32:0/2413946111 >> v1:172.21.15.32:6789/0 conn(0x7f48dc0725e0 legacy=0x7f48dc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:07.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.547+0000 7f48db7fe640 1 -- 172.21.15.32:0/2413946111 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc15da50 con 0x7f48dc15a420 2024-03-16T20:09:07.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.547+0000 7f48daffd640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:09:07.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.555+0000 7f48db7fe640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc15a850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009d90 tx=0x7f48d401c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e3a24df9de310c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48d9ffb640 1 -- 172.21.15.32:0/2413946111 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a5970 con 0x7f48dc15a420 2024-03-16T20:09:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48d9ffb640 1 -- 172.21.15.32:0/2413946111 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40a5ad0 con 0x7f48dc15a420 2024-03-16T20:09:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48d9ffb640 1 -- 172.21.15.32:0/2413946111 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a5e60 con 0x7f48dc15a420 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 -- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 msgr2=0x7f48dc15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc15a850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009d90 tx=0x7f48d401c8b0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 -- 172.21.15.32:0/2413946111 shutdown_connections 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc15a850 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 --2- 172.21.15.32:0/2413946111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 -- 172.21.15.32:0/2413946111 >> 172.21.15.32:0/2413946111 conn(0x7f48dc06c0d0 msgr2=0x7f48dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 -- 172.21.15.32:0/2413946111 shutdown_connections 2024-03-16T20:09:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.563+0000 7f48e1a46640 1 -- 172.21.15.32:0/2413946111 wait complete. 2024-03-16T20:09:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 Processor -- start 2024-03-16T20:09:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 -- start start 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc14ac60 con 0x7f48dc0725e0 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc148ef0 con 0x7f48dc15a420 2024-03-16T20:09:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48e1a46640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc148c10 con 0x7f48dc150d70 2024-03-16T20:09:07.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48daffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.567+0000 7f48da7fc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.571+0000 7f48daffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33124/0 (socket says 172.21.15.32:33124) 2024-03-16T20:09:07.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.571+0000 7f48daffd640 1 -- 172.21.15.32:0/971503999 learned_addr learned my addr 172.21.15.32:0/971503999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:07.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.571+0000 7f48db7fe640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.575+0000 7f48daffd640 1 -- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 msgr2=0x7f48dc145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.575+0000 7f48daffd640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.575+0000 7f48daffd640 1 -- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 msgr2=0x7f48dc149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.575+0000 7f48daffd640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.575+0000 7f48daffd640 1 -- 172.21.15.32:0/971503999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc009420 con 0x7f48dc0725e0 2024-03-16T20:09:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.579+0000 7f48daffd640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ebe0 tx=0x7f48cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.583+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc00eee0 con 0x7f48dc0725e0 2024-03-16T20:09:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.583+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc00cc30 con 0x7f48dc0725e0 2024-03-16T20:09:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.583+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc013470 con 0x7f48dc0725e0 2024-03-16T20:09:07.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.583+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d4009a70 con 0x7f48dc0725e0 2024-03-16T20:09:07.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.583+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc15d560 con 0x7f48dc0725e0 2024-03-16T20:09:07.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.587+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc068000 con 0x7f48dc0725e0 2024-03-16T20:09:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.619+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f48cc0135d0 con 0x7f48dc0725e0 2024-03-16T20:09:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.619+0000 7f48c3fff640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 0x7f48c8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.619+0000 7f48da7fc640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 0x7f48c8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.619+0000 7f48da7fc640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 0x7f48c8069330 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f48d0007fd0 tx=0x7f48d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:07.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.623+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f48cc056070 con 0x7f48dc0725e0 2024-03-16T20:09:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:07.911+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f48cc052310 con 0x7f48dc0725e0 2024-03-16T20:09:08.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.303+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48dc0c3680 con 0x7f48dc0725e0 2024-03-16T20:09:08.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.323+0000 7f48c3fff640 1 -- 172.21.15.32:0/971503999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f48cc021090 con 0x7f48dc0725e0 2024-03-16T20:09:08.316 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:08.316 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:08.316 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:08.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 msgr2=0x7f48c8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 0x7f48c8069330 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f48d0007fd0 tx=0x7f48d0009040 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 msgr2=0x7f48dc0c4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ebe0 tx=0x7f48cc00c520 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 shutdown_connections 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f48c8066e70 0x7f48c8069330 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc15a420 0x7f48dc149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f48dc150d70 0x7f48dc145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 --2- 172.21.15.32:0/971503999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48dc0725e0 0x7f48dc0c4ff0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 >> 172.21.15.32:0/971503999 conn(0x7f48dc06c0d0 msgr2=0x7f48dc06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 shutdown_connections 2024-03-16T20:09:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:08.327+0000 7f48e1a46640 1 -- 172.21.15.32:0/971503999 wait complete. 2024-03-16T20:09:08.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:08.938 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:09.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 Processor -- start 2024-03-16T20:09:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 -- start start 2024-03-16T20:09:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f479c0cc880 0x7f479c06b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f479c0ccc60 0x7f479c153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c15abb0 con 0x7f479c06b890 2024-03-16T20:09:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c1546d0 con 0x7f479c0ccc60 2024-03-16T20:09:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.067+0000 7f47a3492640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c06bde0 con 0x7f479c0cc880 2024-03-16T20:09:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f479c0ccc60 0x7f479c153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f479c0ccc60 0x7f479c153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36140/0 (socket says 172.21.15.32:36140) 2024-03-16T20:09:10.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 -- 172.21.15.32:0/4177004072 learned_addr learned my addr 172.21.15.32:0/4177004072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:10.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a122e640 1 --2- 172.21.15.32:0/4177004072 >> v2:172.21.15.59:3300/0 conn(0x7f479c0cc880 0x7f479c06b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 -- 172.21.15.32:0/4177004072 >> v2:172.21.15.59:3300/0 conn(0x7f479c0cc880 msgr2=0x7f479c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:10.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 --2- 172.21.15.32:0/4177004072 >> v2:172.21.15.59:3300/0 conn(0x7f479c0cc880 0x7f479c06b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 -- 172.21.15.32:0/4177004072 >> v1:172.21.15.32:6789/0 conn(0x7f479c06b890 legacy=0x7f479c15d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:10.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.071+0000 7f47a0a2d640 1 -- 172.21.15.32:0/4177004072 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c15a280 con 0x7f479c0ccc60 2024-03-16T20:09:10.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.075+0000 7f47a0a2d640 1 --2- 172.21.15.32:0/4177004072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c0ccc60 0x7f479c153aa0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f479c0ca230 tx=0x7f478c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8478779370055e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f4793fff640 1 -- 172.21.15.32:0/4177004072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c0aa610 con 0x7f479c0ccc60 2024-03-16T20:09:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f4793fff640 1 -- 172.21.15.32:0/4177004072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478c0a6040 con 0x7f479c0ccc60 2024-03-16T20:09:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f4793fff640 1 -- 172.21.15.32:0/4177004072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c0aa990 con 0x7f479c0ccc60 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 -- 172.21.15.32:0/4177004072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c0ccc60 msgr2=0x7f479c153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 --2- 172.21.15.32:0/4177004072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c0ccc60 0x7f479c153aa0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f479c0ca230 tx=0x7f478c01c4b0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 -- 172.21.15.32:0/4177004072 shutdown_connections 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 --2- 172.21.15.32:0/4177004072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c0ccc60 0x7f479c153aa0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 --2- 172.21.15.32:0/4177004072 >> v2:172.21.15.59:3300/0 conn(0x7f479c0cc880 0x7f479c06b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 -- 172.21.15.32:0/4177004072 >> 172.21.15.32:0/4177004072 conn(0x7f479c06e6c0 msgr2=0x7f479c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 -- 172.21.15.32:0/4177004072 shutdown_connections 2024-03-16T20:09:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.083+0000 7f47a3492640 1 -- 172.21.15.32:0/4177004072 wait complete. 2024-03-16T20:09:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 Processor -- start 2024-03-16T20:09:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 -- start start 2024-03-16T20:09:10.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f479c0cc880 0x7f479c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 0x7f479c148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c148160 con 0x7f479c06b890 2024-03-16T20:09:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c148720 con 0x7f479c0cc880 2024-03-16T20:09:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a3492640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c1573e0 con 0x7f479c0ccc60 2024-03-16T20:09:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a122e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a0a2d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f479c0cc880 0x7f479c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a122e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36144/0 (socket says 172.21.15.32:36144) 2024-03-16T20:09:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.087+0000 7f47a122e640 1 -- 172.21.15.32:0/2886837602 learned_addr learned my addr 172.21.15.32:0/2886837602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a1a2f640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 0x7f479c148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a122e640 1 -- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 msgr2=0x7f479c148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a122e640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 0x7f479c148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a122e640 1 -- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f479c0cc880 msgr2=0x7f479c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a122e640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f479c0cc880 0x7f479c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a122e640 1 -- 172.21.15.32:0/2886837602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0094d0 con 0x7f479c06b890 2024-03-16T20:09:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.091+0000 7f47a1a2f640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 0x7f479c148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:09:10.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.095+0000 7f47a122e640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f479800e930 tx=0x7f479800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:10.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.099+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798009690 con 0x7f479c06b890 2024-03-16T20:09:10.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.099+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800ca00 con 0x7f479c06b890 2024-03-16T20:09:10.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.099+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479800cd70 con 0x7f479c06b890 2024-03-16T20:09:10.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.099+0000 7f47a3492640 1 -- 172.21.15.32:0/2886837602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c1576c0 con 0x7f479c06b890 2024-03-16T20:09:10.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.099+0000 7f47a3492640 1 -- 172.21.15.32:0/2886837602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c154cf0 con 0x7f479c06b890 2024-03-16T20:09:10.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.103+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47780052a0 con 0x7f479c06b890 2024-03-16T20:09:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.387+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4798013470 con 0x7f479c06b890 2024-03-16T20:09:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.387+0000 7f4791ffb640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 0x7f4780069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.387+0000 7f47a0a2d640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 0x7f4780069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:10.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.387+0000 7f47a0a2d640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 0x7f4780069300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f478c0ae000 tx=0x7f478c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:10.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.407+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f47980560a0 con 0x7f479c06b890 2024-03-16T20:09:10.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.411+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4798052340 con 0x7f479c06b890 2024-03-16T20:09:10.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.623+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4778005900 con 0x7f479c06b890 2024-03-16T20:09:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.643+0000 7f4791ffb640 1 -- 172.21.15.32:0/2886837602 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4778005900 con 0x7f479c06b890 2024-03-16T20:09:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:10.636 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 msgr2=0x7f4780069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 0x7f4780069300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f478c0ae000 tx=0x7f478c0b5040 comp rx=0 tx=0).stop 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 msgr2=0x7f479c1f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f479800e930 tx=0x7f479800edf0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 shutdown_connections 2024-03-16T20:09:10.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4780066e40 0x7f4780069300 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f479c0ccc60 0x7f479c148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f479c0cc880 0x7f479c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 --2- 172.21.15.32:0/2886837602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c06b890 0x7f479c1f7d00 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 >> 172.21.15.32:0/2886837602 conn(0x7f479c06e6c0 msgr2=0x7f479c0722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 shutdown_connections 2024-03-16T20:09:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:10.647+0000 7f477f7fe640 1 -- 172.21.15.32:0/2886837602 wait complete. 2024-03-16T20:09:11.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:11.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:12.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:12.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 Processor -- start 2024-03-16T20:09:12.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 -- start start 2024-03-16T20:09:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdce0072c70 0x7fdce00c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce006e6a0 con 0x7fdce006bae0 2024-03-16T20:09:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce006c250 con 0x7fdce0073050 2024-03-16T20:09:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce6cba640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce0156f20 con 0x7fdce0072c70 2024-03-16T20:09:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdce4a56640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdce0072c70 0x7fdce00c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdcdffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdce0073050 0x7fdce0150c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:54134/0 (socket says 172.21.15.32:54134) 2024-03-16T20:09:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.383+0000 7fdcdffff640 1 -- 172.21.15.32:0/841188855 learned_addr learned my addr 172.21.15.32:0/841188855 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.387+0000 7fdce4a56640 1 -- 172.21.15.32:0/841188855 >> v1:172.21.15.59:6789/0 conn(0x7fdce0073050 legacy=0x7fdce0150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.387+0000 7fdce4a56640 1 -- 172.21.15.32:0/841188855 >> v1:172.21.15.32:6789/0 conn(0x7fdce006bae0 legacy=0x7fdce01617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.387+0000 7fdce4a56640 1 -- 172.21.15.32:0/841188855 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce006c510 con 0x7fdce0072c70 2024-03-16T20:09:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.391+0000 7fdce4a56640 1 --2- 172.21.15.32:0/841188855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce00c0f90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40097c0 tx=0x7fdcd401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7d1bd17566f20b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:12.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.395+0000 7fdcdf7fe640 1 -- 172.21.15.32:0/841188855 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40ab650 con 0x7fdce0072c70 2024-03-16T20:09:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdcdf7fe640 1 -- 172.21.15.32:0/841188855 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40ab7b0 con 0x7fdce0072c70 2024-03-16T20:09:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdcdf7fe640 1 -- 172.21.15.32:0/841188855 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40abb40 con 0x7fdce0072c70 2024-03-16T20:09:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 -- 172.21.15.32:0/841188855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 msgr2=0x7fdce00c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 --2- 172.21.15.32:0/841188855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce00c0f90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40097c0 tx=0x7fdcd401c430 comp rx=0 tx=0).stop 2024-03-16T20:09:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 -- 172.21.15.32:0/841188855 shutdown_connections 2024-03-16T20:09:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 --2- 172.21.15.32:0/841188855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce00c0f90 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 -- 172.21.15.32:0/841188855 >> 172.21.15.32:0/841188855 conn(0x7fdce006afc0 msgr2=0x7fdce006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:12.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 -- 172.21.15.32:0/841188855 shutdown_connections 2024-03-16T20:09:12.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.399+0000 7fdce6cba640 1 -- 172.21.15.32:0/841188855 wait complete. 2024-03-16T20:09:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 Processor -- start 2024-03-16T20:09:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 -- start start 2024-03-16T20:09:12.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdce006bae0 0x7fdce006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdce0073050 0x7fdce00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00c4c60 con 0x7fdce0072c70 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00c3d70 con 0x7fdce0073050 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce6cba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00c4050 con 0x7fdce006bae0 2024-03-16T20:09:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:12.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36158/0 (socket says 172.21.15.32:36158) 2024-03-16T20:09:12.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdcdffff640 1 -- 172.21.15.32:0/639811493 learned_addr learned my addr 172.21.15.32:0/639811493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:12.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce5257640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdce0073050 0x7fdce00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:12.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.403+0000 7fdce4a56640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdce006bae0 0x7fdce006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.407+0000 7fdcdffff640 1 -- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdce006bae0 msgr2=0x7fdce006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.407+0000 7fdcdffff640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdce006bae0 0x7fdce006d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.407+0000 7fdcdffff640 1 -- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdce0073050 msgr2=0x7fdce00ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.411+0000 7fdcdffff640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdce0073050 0x7fdce00ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.411+0000 7fdcdffff640 1 -- 172.21.15.32:0/639811493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4009420 con 0x7fdce0072c70 2024-03-16T20:09:12.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.411+0000 7fdcdffff640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00c340 tx=0x7fdccc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.415+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc005cf0 con 0x7fdce0072c70 2024-03-16T20:09:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.415+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc005e50 con 0x7fdce0072c70 2024-03-16T20:09:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.415+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0072f0 con 0x7fdce0072c70 2024-03-16T20:09:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.419+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce00c4330 con 0x7fdce0072c70 2024-03-16T20:09:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.419+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce00c48f0 con 0x7fdce0072c70 2024-03-16T20:09:12.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.419+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce00c4ab0 con 0x7fdce0072c70 2024-03-16T20:09:12.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.455+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdccc007450 con 0x7fdce0072c70 2024-03-16T20:09:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.455+0000 7fdcdd7fa640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 0x7fdcc4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.455+0000 7fdce4a56640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 0x7fdcc4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:12.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.455+0000 7fdce4a56640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 0x7fdcc4069350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40af000 tx=0x7fdcd40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.459+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdccc054f80 con 0x7fdce0072c70 2024-03-16T20:09:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.739+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdccc051220 con 0x7fdce0072c70 2024-03-16T20:09:12.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:09:12.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:09:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.947+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce0068000 con 0x7fdce0072c70 2024-03-16T20:09:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.959+0000 7fdcdd7fa640 1 -- 172.21.15.32:0/639811493 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fdccc017090 con 0x7fdce0072c70 2024-03-16T20:09:12.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:12.952 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:12.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 msgr2=0x7fdcc4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 0x7fdcc4069350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40af000 tx=0x7fdcd40a7000 comp rx=0 tx=0).stop 2024-03-16T20:09:12.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 msgr2=0x7fdce0147410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00c340 tx=0x7fdccc00c800 comp rx=0 tx=0).stop 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 shutdown_connections 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdcc4066e90 0x7fdcc4069350 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdce0073050 0x7fdce00ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdce0072c70 0x7fdce0147410 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 --2- 172.21.15.32:0/639811493 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdce006bae0 0x7fdce006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 >> 172.21.15.32:0/639811493 conn(0x7fdce006afc0 msgr2=0x7fdce01488c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 shutdown_connections 2024-03-16T20:09:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:12.963+0000 7fdce6cba640 1 -- 172.21.15.32:0/639811493 wait complete. 2024-03-16T20:09:13.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:13.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:14.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 Processor -- start 2024-03-16T20:09:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 -- start start 2024-03-16T20:09:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 0x7fc1440c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc144158c80 con 0x7fc1440c81e0 2024-03-16T20:09:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14415bfc0 con 0x7fc144151ef0 2024-03-16T20:09:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14bf75640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440b0ad0 con 0x7fc14415b9b0 2024-03-16T20:09:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc149510640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1440c81e0 0x7fc14415b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51202/0 (socket says 172.21.15.32:51202) 2024-03-16T20:09:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc149510640 1 -- 172.21.15.32:0/3167445215 learned_addr learned my addr 172.21.15.32:0/3167445215 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.695+0000 7fc14a512640 1 --2- 172.21.15.32:0/3167445215 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 0x7fc1440c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:14.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.699+0000 7fc148d0f640 1 -- 172.21.15.32:0/3167445215 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284419320 0 0) 0x7fc144158c80 con 0x7fc1440c81e0 2024-03-16T20:09:14.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.699+0000 7fc148d0f640 1 -- 172.21.15.32:0/3167445215 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc130003530 con 0x7fc1440c81e0 2024-03-16T20:09:14.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.703+0000 7fc14a512640 1 -- 172.21.15.32:0/3167445215 >> v1:172.21.15.59:6789/0 conn(0x7fc144151ef0 legacy=0x7fc1440c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:14.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.703+0000 7fc14a512640 1 -- 172.21.15.32:0/3167445215 >> v1:172.21.15.32:6789/0 conn(0x7fc1440c81e0 legacy=0x7fc14415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:14.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.703+0000 7fc14a512640 1 -- 172.21.15.32:0/3167445215 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144157ec0 con 0x7fc14415b9b0 2024-03-16T20:09:14.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.707+0000 7fc14a512640 1 --2- 172.21.15.32:0/3167445215 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 0x7fc1440c8630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009dc0 tx=0x7fc13c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39b25b92f29477c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc148d0f640 1 -- 172.21.15.32:0/3167445215 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0af600 con 0x7fc14415b9b0 2024-03-16T20:09:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc148d0f640 1 -- 172.21.15.32:0/3167445215 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0ab050 con 0x7fc14415b9b0 2024-03-16T20:09:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc148d0f640 1 -- 172.21.15.32:0/3167445215 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0af980 con 0x7fc14415b9b0 2024-03-16T20:09:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 -- 172.21.15.32:0/3167445215 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 msgr2=0x7fc1440c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 --2- 172.21.15.32:0/3167445215 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 0x7fc1440c8630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009dc0 tx=0x7fc13c01f170 comp rx=0 tx=0).stop 2024-03-16T20:09:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 -- 172.21.15.32:0/3167445215 shutdown_connections 2024-03-16T20:09:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 --2- 172.21.15.32:0/3167445215 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc14415b9b0 0x7fc1440c8630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 -- 172.21.15.32:0/3167445215 >> 172.21.15.32:0/3167445215 conn(0x7fc144070f90 msgr2=0x7fc1440713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 -- 172.21.15.32:0/3167445215 shutdown_connections 2024-03-16T20:09:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.711+0000 7fc14bf75640 1 -- 172.21.15.32:0/3167445215 wait complete. 2024-03-16T20:09:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 Processor -- start 2024-03-16T20:09:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 -- start start 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1440c81e0 0x7fc14414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc144151ef0 0x7fc1440c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14414f160 con 0x7fc14415b9b0 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14414eba0 con 0x7fc1440c81e0 2024-03-16T20:09:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14bf75640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14414ee80 con 0x7fc144151ef0 2024-03-16T20:09:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc149d11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1440c81e0 0x7fc14414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14a512640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14a512640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36172/0 (socket says 172.21.15.32:36172) 2024-03-16T20:09:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc14a512640 1 -- 172.21.15.32:0/1892172453 learned_addr learned my addr 172.21.15.32:0/1892172453 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:14.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.715+0000 7fc149510640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc144151ef0 0x7fc1440c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:14.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.719+0000 7fc14a512640 1 -- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc144151ef0 msgr2=0x7fc1440c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:14.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.719+0000 7fc14a512640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc144151ef0 0x7fc1440c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:14.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.719+0000 7fc14a512640 1 -- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1440c81e0 msgr2=0x7fc14414e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:14.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.719+0000 7fc14a512640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1440c81e0 0x7fc14414e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:14.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.719+0000 7fc14a512640 1 -- 172.21.15.32:0/1892172453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c009a70 con 0x7fc14415b9b0 2024-03-16T20:09:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.723+0000 7fc14a512640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc13c01bd80 tx=0x7fc13c01bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c009450 con 0x7fc14415b9b0 2024-03-16T20:09:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0b9410 con 0x7fc14415b9b0 2024-03-16T20:09:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0b97c0 con 0x7fc14415b9b0 2024-03-16T20:09:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1440b3190 con 0x7fc14415b9b0 2024-03-16T20:09:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1440b1730 con 0x7fc14415b9b0 2024-03-16T20:09:14.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:14.727+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc144068000 con 0x7fc14415b9b0 2024-03-16T20:09:14.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.007+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc13c0b6050 con 0x7fc14415b9b0 2024-03-16T20:09:15.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.007+0000 7fc13a7fc640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 0x7fc128069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:15.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.011+0000 7fc149d11640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 0x7fc128069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:15.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.011+0000 7fc149d11640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 0x7fc128069380 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc140001810 tx=0x7fc140009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:15.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.035+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc13c0f44d0 con 0x7fc14415b9b0 2024-03-16T20:09:15.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.035+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc13c0f0770 con 0x7fc14415b9b0 2024-03-16T20:09:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.315+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1440b3640 con 0x7fc14415b9b0 2024-03-16T20:09:15.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.327+0000 7fc13a7fc640 1 -- 172.21.15.32:0/1892172453 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc13c0b4070 con 0x7fc14415b9b0 2024-03-16T20:09:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:15.321 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 msgr2=0x7fc128069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 0x7fc128069380 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc140001810 tx=0x7fc140009040 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 msgr2=0x7fc1441f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc13c01bd80 tx=0x7fc13c01bf10 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 shutdown_connections 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc128066ec0 0x7fc128069380 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14415b9b0 0x7fc1441f0bd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc144151ef0 0x7fc1440c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 --2- 172.21.15.32:0/1892172453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1440c81e0 0x7fc14414e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 >> 172.21.15.32:0/1892172453 conn(0x7fc144070f90 msgr2=0x7fc1440c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 shutdown_connections 2024-03-16T20:09:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:15.331+0000 7fc14bf75640 1 -- 172.21.15.32:0/1892172453 wait complete. 2024-03-16T20:09:15.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:15.950 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:16.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 Processor -- start 2024-03-16T20:09:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 -- start start 2024-03-16T20:09:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66fc071110 0x7f66fc0caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f66fc0714f0 0x7f66fc0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0c9100 0x7f66fc153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0c9d10 con 0x7f66fc071110 2024-03-16T20:09:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0c9590 con 0x7f66fc0714f0 2024-03-16T20:09:17.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f67031b4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0ca8e0 con 0x7f66fc0c9100 2024-03-16T20:09:17.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f6700f50640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66fc071110 0x7f66fc0caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f6700f50640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66fc071110 0x7f66fc0caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36186/0 (socket says 172.21.15.32:36186) 2024-03-16T20:09:17.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f6700f50640 1 -- 172.21.15.32:0/3121996829 learned_addr learned my addr 172.21.15.32:0/3121996829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:17.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f6701751640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0c9100 0x7f66fc153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.071+0000 7f66fbfff640 1 --2- 172.21.15.32:0/3121996829 >> v2:172.21.15.59:3300/0 conn(0x7f66fc0714f0 0x7f66fc0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.075+0000 7f6700f50640 1 -- 172.21.15.32:0/3121996829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0c9100 msgr2=0x7f66fc153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.075+0000 7f6700f50640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0c9100 0x7f66fc153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.075+0000 7f6700f50640 1 -- 172.21.15.32:0/3121996829 >> v2:172.21.15.59:3300/0 conn(0x7f66fc0714f0 msgr2=0x7f66fc0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.075+0000 7f6700f50640 1 --2- 172.21.15.32:0/3121996829 >> v2:172.21.15.59:3300/0 conn(0x7f66fc0714f0 0x7f66fc0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.075+0000 7f6700f50640 1 -- 172.21.15.32:0/3121996829 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc0cab80 con 0x7f66fc071110 2024-03-16T20:09:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.079+0000 7f6700f50640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc071110 0x7f66fc0caf70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0ce990 tx=0x7f66f001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7f70215f3afecc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:17.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f66fb7fe640 1 -- 172.21.15.32:0/3121996829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00ac620 con 0x7f66fc071110 2024-03-16T20:09:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f66fb7fe640 1 -- 172.21.15.32:0/3121996829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f00ac780 con 0x7f66fc071110 2024-03-16T20:09:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f66fb7fe640 1 -- 172.21.15.32:0/3121996829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00acb10 con 0x7f66fc071110 2024-03-16T20:09:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 -- 172.21.15.32:0/3121996829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc071110 msgr2=0x7f66fc0caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc071110 0x7f66fc0caf70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0ce990 tx=0x7f66f001c4f0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 -- 172.21.15.32:0/3121996829 shutdown_connections 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0c9100 0x7f66fc153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 --2- 172.21.15.32:0/3121996829 >> v2:172.21.15.59:3300/0 conn(0x7f66fc0714f0 0x7f66fc0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 --2- 172.21.15.32:0/3121996829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc071110 0x7f66fc0caf70 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 -- 172.21.15.32:0/3121996829 >> 172.21.15.32:0/3121996829 conn(0x7f66fc06afc0 msgr2=0x7f66fc06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 -- 172.21.15.32:0/3121996829 shutdown_connections 2024-03-16T20:09:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.083+0000 7f67031b4640 1 -- 172.21.15.32:0/3121996829 wait complete. 2024-03-16T20:09:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 Processor -- start 2024-03-16T20:09:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 -- start start 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66fc071110 0x7f66fc14b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0714f0 0x7f66fc157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc156e30 con 0x7f66fc0c9100 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc14fb10 con 0x7f66fc071110 2024-03-16T20:09:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f67031b4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc155ca0 con 0x7f66fc0714f0 2024-03-16T20:09:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f6701751640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f6701751640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36202/0 (socket says 172.21.15.32:36202) 2024-03-16T20:09:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.087+0000 7f6701751640 1 -- 172.21.15.32:0/142833035 learned_addr learned my addr 172.21.15.32:0/142833035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6700f50640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66fc071110 0x7f66fc14b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f66fbfff640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0714f0 0x7f66fc157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6701751640 1 -- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0714f0 msgr2=0x7f66fc157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6701751640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0714f0 0x7f66fc157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6701751640 1 -- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66fc071110 msgr2=0x7f66fc14b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6701751640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66fc071110 0x7f66fc14b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.091+0000 7f6701751640 1 -- 172.21.15.32:0/142833035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00094d0 con 0x7f66fc0c9100 2024-03-16T20:09:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.095+0000 7f6701751640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009d30 tx=0x7f66f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4009ba0 con 0x7f66fc0c9100 2024-03-16T20:09:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f400f040 con 0x7f66fc0c9100 2024-03-16T20:09:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc155f40 con 0x7f66fc0c9100 2024-03-16T20:09:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f40146c0 con 0x7f66fc0c9100 2024-03-16T20:09:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc156500 con 0x7f66fc0c9100 2024-03-16T20:09:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.099+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc068000 con 0x7f66fc0c9100 2024-03-16T20:09:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.131+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f66f4014820 con 0x7f66fc0c9100 2024-03-16T20:09:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.131+0000 7f66f97fa640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 0x7f66e0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.131+0000 7f6700f50640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 0x7f66e0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:17.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.135+0000 7f6700f50640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 0x7f66e0069350 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f66f00ae000 tx=0x7f66f00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.139+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f66f4056870 con 0x7f66fc0c9100 2024-03-16T20:09:17.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.411+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f66f4052b10 con 0x7f66fc0c9100 2024-03-16T20:09:17.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.615+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66fc0ce270 con 0x7f66fc0c9100 2024-03-16T20:09:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.631+0000 7f66f97fa640 1 -- 172.21.15.32:0/142833035 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f66f4018070 con 0x7f66fc0c9100 2024-03-16T20:09:17.624 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:17.624 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 msgr2=0x7f66e0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 0x7f66e0069350 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f66f00ae000 tx=0x7f66f00a7000 comp rx=0 tx=0).stop 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 msgr2=0x7f66fc150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009d30 tx=0x7f66f400c490 comp rx=0 tx=0).stop 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 shutdown_connections 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f66e0066e90 0x7f66e0069350 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66fc0c9100 0x7f66fc150120 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f66fc0714f0 0x7f66fc157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 --2- 172.21.15.32:0/142833035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66fc071110 0x7f66fc14b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 >> 172.21.15.32:0/142833035 conn(0x7f66fc06afc0 msgr2=0x7f66fc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 shutdown_connections 2024-03-16T20:09:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:17.635+0000 7f67031b4640 1 -- 172.21.15.32:0/142833035 wait complete. 2024-03-16T20:09:18.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:18.253 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:19.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 Processor -- start 2024-03-16T20:09:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 -- start start 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ccc154350 0x7f1ccc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc0c2d70 0x7f1ccc15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1ccc072a10 0x7f1ccc072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ccc06bae0 con 0x7f1ccc154350 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ccc15a540 con 0x7f1ccc072a10 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.371+0000 7f1cd2ca8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc0c7fd0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ccc154350 0x7f1ccc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ccc154350 0x7f1ccc06c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48698/0 (socket says 172.21.15.32:48698) 2024-03-16T20:09:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 -- 172.21.15.32:0/1919877401 learned_addr learned my addr 172.21.15.32:0/1919877401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1ccbfff640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc0c2d70 0x7f1ccc15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd1245640 1 --2- 172.21.15.32:0/1919877401 >> v2:172.21.15.59:3300/0 conn(0x7f1ccc072a10 0x7f1ccc072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 -- 172.21.15.32:0/1919877401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc0c2d70 msgr2=0x7f1ccc15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc0c2d70 0x7f1ccc15a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 -- 172.21.15.32:0/1919877401 >> v2:172.21.15.59:3300/0 conn(0x7f1ccc072a10 msgr2=0x7f1ccc072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1919877401 >> v2:172.21.15.59:3300/0 conn(0x7f1ccc072a10 0x7f1ccc072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.375+0000 7f1cd0a44640 1 -- 172.21.15.32:0/1919877401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc06bd10 con 0x7f1ccc154350 2024-03-16T20:09:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.379+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc154350 0x7f1ccc06c250 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc06ca80 tx=0x7f1cc001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690dfeb10761e62b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1ccb7fe640 1 -- 172.21.15.32:0/1919877401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00ab600 con 0x7f1ccc154350 2024-03-16T20:09:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1ccb7fe640 1 -- 172.21.15.32:0/1919877401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc00a7050 con 0x7f1ccc154350 2024-03-16T20:09:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1ccb7fe640 1 -- 172.21.15.32:0/1919877401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00ab980 con 0x7f1ccc154350 2024-03-16T20:09:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1919877401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc154350 msgr2=0x7f1ccc06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc154350 0x7f1ccc06c250 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc06ca80 tx=0x7f1cc001c470 comp rx=0 tx=0).stop 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1919877401 shutdown_connections 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1919877401 >> v2:172.21.15.59:3300/0 conn(0x7f1ccc072a10 0x7f1ccc072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc0c2d70 0x7f1ccc15a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1919877401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc154350 0x7f1ccc06c250 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1919877401 >> 172.21.15.32:0/1919877401 conn(0x7f1ccc06afc0 msgr2=0x7f1ccc06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1919877401 shutdown_connections 2024-03-16T20:09:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1919877401 wait complete. 2024-03-16T20:09:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 Processor -- start 2024-03-16T20:09:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- start start 2024-03-16T20:09:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc072a10 0x7f1ccc1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ccc154350 0x7f1ccc151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc1f38d0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc157440 con 0x7f1ccc154350 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.387+0000 7f1cd2ca8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc155af0 con 0x7f1ccc072a10 2024-03-16T20:09:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.391+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.391+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48714/0 (socket says 172.21.15.32:48714) 2024-03-16T20:09:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.391+0000 7f1ccbfff640 1 -- 172.21.15.32:0/1816576742 learned_addr learned my addr 172.21.15.32:0/1816576742 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.391+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc072a10 0x7f1ccc1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.391+0000 7f1cd1245640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ccc154350 0x7f1ccc151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.395+0000 7f1ccbfff640 1 -- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc072a10 msgr2=0x7f1ccc1f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.395+0000 7f1ccbfff640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc072a10 0x7f1ccc1f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.395+0000 7f1ccbfff640 1 -- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ccc154350 msgr2=0x7f1ccc151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.395+0000 7f1ccbfff640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ccc154350 0x7f1ccc151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.395+0000 7f1ccbfff640 1 -- 172.21.15.32:0/1816576742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc00094d0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.399+0000 7f1ccbfff640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800e930 tx=0x7f1cb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8009690 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb800ca00 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc157700 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800cd70 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc154db0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.403+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc0674e0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.439+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1cb8013470 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.439+0000 7f1cc97fa640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 0x7f1cb0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:19.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.439+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 0x7f1cb0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:19.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.439+0000 7f1cd0a44640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 0x7f1cb0069300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00ad000 tx=0x7f1cc00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:19.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.727+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1cb80560f0 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.727+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1cb8052390 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.931+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ccc068000 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cc97fa640 1 -- 172.21.15.32:0/1816576742 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1cb8021020 con 0x7f1ccc0c2d70 2024-03-16T20:09:19.989 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:19.990 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:19.990 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 msgr2=0x7f1cb0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 0x7f1cb0069300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00ad000 tx=0x7f1cc00a6000 comp rx=0 tx=0).stop 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 msgr2=0x7f1ccc1555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800e930 tx=0x7f1cb800edf0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 shutdown_connections 2024-03-16T20:09:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1cb0066e40 0x7f1cb0069300 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ccc154350 0x7f1ccc151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0c2d70 0x7f1ccc1555b0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 --2- 172.21.15.32:0/1816576742 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ccc072a10 0x7f1ccc1f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 >> 172.21.15.32:0/1816576742 conn(0x7f1ccc06afc0 msgr2=0x7f1ccc066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 shutdown_connections 2024-03-16T20:09:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:19.999+0000 7f1cd2ca8640 1 -- 172.21.15.32:0/1816576742 wait complete. 2024-03-16T20:09:20.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:20.613 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:21.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:21.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 Processor -- start 2024-03-16T20:09:21.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 -- start start 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 0x7fc79c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c158c80 con 0x7fc79c151ef0 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c0b0ad0 con 0x7fc79c0c81e0 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.731+0000 7fc7a189b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c157cb0 con 0x7fc79c15b2a0 2024-03-16T20:09:21.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc79affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc79c151ef0 0x7fc79c0c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40876/0 (socket says 172.21.15.32:40876) 2024-03-16T20:09:21.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc79affd640 1 -- 172.21.15.32:0/1948892678 learned_addr learned my addr 172.21.15.32:0/1948892678 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:21.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc79b7fe640 1 --2- 172.21.15.32:0/1948892678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc79a7fc640 1 --2- 172.21.15.32:0/1948892678 >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 0x7fc79c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc799ffb640 1 -- 172.21.15.32:0/1948892678 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539699334 0 0) 0x7fc79c158c80 con 0x7fc79c151ef0 2024-03-16T20:09:21.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.735+0000 7fc799ffb640 1 -- 172.21.15.32:0/1948892678 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc784003530 con 0x7fc79c151ef0 2024-03-16T20:09:21.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.739+0000 7fc79b7fe640 1 -- 172.21.15.32:0/1948892678 >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 msgr2=0x7fc79c0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:21.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.739+0000 7fc79b7fe640 1 --2- 172.21.15.32:0/1948892678 >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 0x7fc79c0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:21.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.739+0000 7fc79b7fe640 1 -- 172.21.15.32:0/1948892678 >> v1:172.21.15.32:6789/0 conn(0x7fc79c151ef0 legacy=0x7fc79c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:21.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.739+0000 7fc79b7fe640 1 -- 172.21.15.32:0/1948892678 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c15bfc0 con 0x7fc79c15b2a0 2024-03-16T20:09:21.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.739+0000 7fc79a7fc640 1 --2- 172.21.15.32:0/1948892678 >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 0x7fc79c0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:09:21.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.743+0000 7fc79b7fe640 1 --2- 172.21.15.32:0/1948892678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c15b6a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc794009d90 tx=0x7fc79401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369db5effdb122b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:21.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc799ffb640 1 -- 172.21.15.32:0/1948892678 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940af600 con 0x7fc79c15b2a0 2024-03-16T20:09:21.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc799ffb640 1 -- 172.21.15.32:0/1948892678 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7940ab050 con 0x7fc79c15b2a0 2024-03-16T20:09:21.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc799ffb640 1 -- 172.21.15.32:0/1948892678 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940af980 con 0x7fc79c15b2a0 2024-03-16T20:09:21.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- 172.21.15.32:0/1948892678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 msgr2=0x7fc79c15b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:21.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- 172.21.15.32:0/1948892678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c15b6a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc794009d90 tx=0x7fc79401f150 comp rx=0 tx=0).stop 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- 172.21.15.32:0/1948892678 shutdown_connections 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- 172.21.15.32:0/1948892678 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c15b6a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- 172.21.15.32:0/1948892678 >> v2:172.21.15.59:3300/0 conn(0x7fc79c0c81e0 0x7fc79c0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- 172.21.15.32:0/1948892678 >> 172.21.15.32:0/1948892678 conn(0x7fc79c070f90 msgr2=0x7fc79c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- 172.21.15.32:0/1948892678 shutdown_connections 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- 172.21.15.32:0/1948892678 wait complete. 2024-03-16T20:09:21.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 Processor -- start 2024-03-16T20:09:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- start start 2024-03-16T20:09:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c151ef0 0x7fc79c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c06c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c155c80 con 0x7fc79c0c81e0 2024-03-16T20:09:21.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c151750 con 0x7fc79c151ef0 2024-03-16T20:09:21.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.751+0000 7fc7a189b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c151490 con 0x7fc79c15b2a0 2024-03-16T20:09:21.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.755+0000 7fc79affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.755+0000 7fc79a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c151ef0 0x7fc79c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.755+0000 7fc79affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48728/0 (socket says 172.21.15.32:48728) 2024-03-16T20:09:21.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.755+0000 7fc79affd640 1 -- 172.21.15.32:0/703682994 learned_addr learned my addr 172.21.15.32:0/703682994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.755+0000 7fc79b7fe640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.759+0000 7fc79affd640 1 -- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 msgr2=0x7fc79c06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:21.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.759+0000 7fc79affd640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c06c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:21.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.759+0000 7fc79affd640 1 -- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c151ef0 msgr2=0x7fc79c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:21.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.759+0000 7fc79affd640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c151ef0 0x7fc79c06c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:21.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.759+0000 7fc79affd640 1 -- 172.21.15.32:0/703682994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790009420 con 0x7fc79c0c81e0 2024-03-16T20:09:21.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.763+0000 7fc79affd640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c480 tx=0x7fc78c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c005cf0 con 0x7fc79c0c81e0 2024-03-16T20:09:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c005e50 con 0x7fc79c0c81e0 2024-03-16T20:09:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c007290 con 0x7fc79c0c81e0 2024-03-16T20:09:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc794009a70 con 0x7fc79c0c81e0 2024-03-16T20:09:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c1f05b0 con 0x7fc79c0c81e0 2024-03-16T20:09:21.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.771+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c068000 con 0x7fc79c0c81e0 2024-03-16T20:09:21.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.795+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc78c0073f0 con 0x7fc79c0c81e0 2024-03-16T20:09:21.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.799+0000 7fc783fff640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 0x7fc788069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:21.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.799+0000 7fc79a7fc640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 0x7fc788069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:21.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.799+0000 7fc79a7fc640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 0x7fc788069300 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0c8930 tx=0x7fc790009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:21.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:21.799+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc78c055100 con 0x7fc79c0c81e0 2024-03-16T20:09:22.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.107+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc78c0513a0 con 0x7fc79c0c81e0 2024-03-16T20:09:22.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.307+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc79c070a40 con 0x7fc79c0c81e0 2024-03-16T20:09:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.331+0000 7fc783fff640 1 -- 172.21.15.32:0/703682994 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc78c017070 con 0x7fc79c0c81e0 2024-03-16T20:09:22.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:22.322 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.331+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 msgr2=0x7fc788069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.331+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 0x7fc788069300 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0c8930 tx=0x7fc790009190 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.331+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 msgr2=0x7fc79c1505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.331+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c480 tx=0x7fc78c00c940 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 shutdown_connections 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc788066e40 0x7fc788069300 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc79c15b2a0 0x7fc79c06c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c151ef0 0x7fc79c06c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 --2- 172.21.15.32:0/703682994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c0c81e0 0x7fc79c1505c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 >> 172.21.15.32:0/703682994 conn(0x7fc79c070f90 msgr2=0x7fc79c156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 shutdown_connections 2024-03-16T20:09:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:22.335+0000 7fc7a189b640 1 -- 172.21.15.32:0/703682994 wait complete. 2024-03-16T20:09:22.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:22.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:23.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 Processor -- start 2024-03-16T20:09:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 -- start start 2024-03-16T20:09:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a3c153fa0 0x7f9a3c070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c0c7470 0x7f9a3c15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0c8590 con 0x7f9a3c15d920 2024-03-16T20:09:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3c0c3f00 con 0x7f9a3c153fa0 2024-03-16T20:09:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.063+0000 7f9a43097640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c070650 con 0x7f9a3c0c7470 2024-03-16T20:09:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.067+0000 7f9a41634640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9a3c15d920 0x7f9a3c0724a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49912/0 (socket says 172.21.15.32:49912) 2024-03-16T20:09:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.067+0000 7f9a41634640 1 -- 172.21.15.32:0/3323486865 learned_addr learned my addr 172.21.15.32:0/3323486865 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.067+0000 7f9a3bfff640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c0c7470 0x7f9a3c15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.067+0000 7f9a40e33640 1 --2- 172.21.15.32:0/3323486865 >> v2:172.21.15.32:3300/0 conn(0x7f9a3c153fa0 0x7f9a3c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.071+0000 7f9a40e33640 1 -- 172.21.15.32:0/3323486865 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c0c7470 msgr2=0x7f9a3c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.071+0000 7f9a40e33640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c0c7470 0x7f9a3c15d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.071+0000 7f9a40e33640 1 -- 172.21.15.32:0/3323486865 >> v1:172.21.15.59:6789/0 conn(0x7f9a3c15d920 legacy=0x7f9a3c0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.071+0000 7f9a40e33640 1 -- 172.21.15.32:0/3323486865 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c073590 con 0x7f9a3c153fa0 2024-03-16T20:09:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.075+0000 7f9a40e33640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c070840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9a300097c0 tx=0x7f9a3001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa4d5eebf21f5db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a3b7fe640 1 -- 172.21.15.32:0/3323486865 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300a8990 con 0x7f9a3c153fa0 2024-03-16T20:09:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a3b7fe640 1 -- 172.21.15.32:0/3323486865 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300a8af0 con 0x7f9a3c153fa0 2024-03-16T20:09:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a3b7fe640 1 -- 172.21.15.32:0/3323486865 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300a8e80 con 0x7f9a3c153fa0 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 -- 172.21.15.32:0/3323486865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c153fa0 msgr2=0x7f9a3c070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c070840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9a300097c0 tx=0x7f9a3001ec90 comp rx=0 tx=0).stop 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 -- 172.21.15.32:0/3323486865 shutdown_connections 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c0c7470 0x7f9a3c15d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 --2- 172.21.15.32:0/3323486865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c070840 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 -- 172.21.15.32:0/3323486865 >> 172.21.15.32:0/3323486865 conn(0x7f9a3c06c0d0 msgr2=0x7f9a3c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 -- 172.21.15.32:0/3323486865 shutdown_connections 2024-03-16T20:09:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.079+0000 7f9a43097640 1 -- 172.21.15.32:0/3323486865 wait complete. 2024-03-16T20:09:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 Processor -- start 2024-03-16T20:09:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 -- start start 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c15d920 0x7f9a3c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c156020 con 0x7f9a3c0c7470 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c1efd20 con 0x7f9a3c153fa0 2024-03-16T20:09:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a43097640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c156a20 con 0x7f9a3c15d920 2024-03-16T20:09:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a40e33640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a41634640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c15d920 0x7f9a3c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a40e33640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48746/0 (socket says 172.21.15.32:48746) 2024-03-16T20:09:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.083+0000 7f9a40e33640 1 -- 172.21.15.32:0/750761882 learned_addr learned my addr 172.21.15.32:0/750761882 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.091+0000 7f9a40e33640 1 -- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c15d920 msgr2=0x7f9a3c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.091+0000 7f9a40e33640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c15d920 0x7f9a3c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.091+0000 7f9a40e33640 1 -- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a3c153fa0 msgr2=0x7f9a3c0cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.091+0000 7f9a40e33640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c0cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.091+0000 7f9a40e33640 1 -- 172.21.15.32:0/750761882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30009420 con 0x7f9a3c0c7470 2024-03-16T20:09:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.095+0000 7f9a40e33640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9a300b3040 tx=0x7f9a3001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300b0e10 con 0x7f9a3c0c7470 2024-03-16T20:09:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300b9410 con 0x7f9a3c0c7470 2024-03-16T20:09:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300b97c0 con 0x7f9a3c0c7470 2024-03-16T20:09:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c156cc0 con 0x7f9a3c0c7470 2024-03-16T20:09:24.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c157280 con 0x7f9a3c0c7470 2024-03-16T20:09:24.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.099+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c068000 con 0x7f9a3c0c7470 2024-03-16T20:09:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.131+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9a300b7040 con 0x7f9a3c0c7470 2024-03-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.131+0000 7f9a397fa640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 0x7f9a20069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.135+0000 7f9a3bfff640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 0x7f9a20069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.135+0000 7f9a3bfff640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 0x7f9a20069330 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c153080 tx=0x7f9a28009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.135+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9a300f3e70 con 0x7f9a3c0c7470 2024-03-16T20:09:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.411+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9a300f0110 con 0x7f9a3c0c7470 2024-03-16T20:09:24.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.615+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a3c072bf0 con 0x7f9a3c0c7470 2024-03-16T20:09:24.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.639+0000 7f9a397fa640 1 -- 172.21.15.32:0/750761882 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f9a300b5070 con 0x7f9a3c0c7470 2024-03-16T20:09:24.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:24.632 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 msgr2=0x7f9a20069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 0x7f9a20069330 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c153080 tx=0x7f9a28009190 comp rx=0 tx=0).stop 2024-03-16T20:09:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 msgr2=0x7f9a3c0c6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9a300b3040 tx=0x7f9a3001ba30 comp rx=0 tx=0).stop 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 shutdown_connections 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9a20066e70 0x7f9a20069330 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9a3c15d920 0x7f9a3c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a3c153fa0 0x7f9a3c0cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 --2- 172.21.15.32:0/750761882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3c0c7470 0x7f9a3c0c6300 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 >> 172.21.15.32:0/750761882 conn(0x7f9a3c06c0d0 msgr2=0x7f9a3c0c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 shutdown_connections 2024-03-16T20:09:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:24.643+0000 7f9a43097640 1 -- 172.21.15.32:0/750761882 wait complete. 2024-03-16T20:09:25.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:25.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:26.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:26.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 Processor -- start 2024-03-16T20:09:26.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 -- start start 2024-03-16T20:09:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5580cdf60 0x7fb558151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 0x7fb558152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb558157eb0 con 0x7fb5580cdb80 2024-03-16T20:09:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580c4c80 con 0x7fb5580cdf60 2024-03-16T20:09:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55d1ad640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806fa80 con 0x7fb5580c7830 2024-03-16T20:09:26.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5580cdf60 0x7fb558151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb55759e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 0x7fb558152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb556d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb5580cdb80 0x7fb55806f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49920/0 (socket says 172.21.15.32:49920) 2024-03-16T20:09:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.375+0000 7fb556d9d640 1 -- 172.21.15.32:0/2047945886 learned_addr learned my addr 172.21.15.32:0/2047945886 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:26.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.383+0000 7fb55759e640 1 -- 172.21.15.32:0/2047945886 >> v2:172.21.15.59:3300/0 conn(0x7fb5580cdf60 msgr2=0x7fb558151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:26.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.383+0000 7fb55759e640 1 --2- 172.21.15.32:0/2047945886 >> v2:172.21.15.59:3300/0 conn(0x7fb5580cdf60 0x7fb558151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:26.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.383+0000 7fb55759e640 1 -- 172.21.15.32:0/2047945886 >> v1:172.21.15.59:6789/0 conn(0x7fb5580cdb80 legacy=0x7fb55806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:26.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.383+0000 7fb55759e640 1 -- 172.21.15.32:0/2047945886 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580c4f40 con 0x7fb5580c7830 2024-03-16T20:09:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.391+0000 7fb55759e640 1 --2- 172.21.15.32:0/2047945886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 0x7fb558152060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb550009d90 tx=0x7fb55001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8674570ce62f2638 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb555d9b640 1 -- 172.21.15.32:0/2047945886 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500a5a00 con 0x7fb5580c7830 2024-03-16T20:09:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb555d9b640 1 -- 172.21.15.32:0/2047945886 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5500a5b60 con 0x7fb5580c7830 2024-03-16T20:09:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb555d9b640 1 -- 172.21.15.32:0/2047945886 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500ad440 con 0x7fb5580c7830 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- 172.21.15.32:0/2047945886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 msgr2=0x7fb558152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/2047945886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 0x7fb558152060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb550009d90 tx=0x7fb55001c940 comp rx=0 tx=0).stop 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- 172.21.15.32:0/2047945886 shutdown_connections 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/2047945886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580c7830 0x7fb558152060 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/2047945886 >> v2:172.21.15.59:3300/0 conn(0x7fb5580cdf60 0x7fb558151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- 172.21.15.32:0/2047945886 >> 172.21.15.32:0/2047945886 conn(0x7fb558066ea0 msgr2=0x7fb5580672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- 172.21.15.32:0/2047945886 shutdown_connections 2024-03-16T20:09:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- 172.21.15.32:0/2047945886 wait complete. 2024-03-16T20:09:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 Processor -- start 2024-03-16T20:09:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.403+0000 7fb55d1ad640 1 -- start start 2024-03-16T20:09:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5580cdb80 0x7fb5580ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580cdf60 0x7fb55806c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806c570 con 0x7fb5580c7830 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581540b0 con 0x7fb5580cdb80 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55d1ad640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581f3a90 con 0x7fb5580cdf60 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb556d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5580cdb80 0x7fb5580ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb556d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48758/0 (socket says 172.21.15.32:48758) 2024-03-16T20:09:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb556d9d640 1 -- 172.21.15.32:0/3784310337 learned_addr learned my addr 172.21.15.32:0/3784310337 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.407+0000 7fb55759e640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580cdf60 0x7fb55806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.411+0000 7fb556d9d640 1 -- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580cdf60 msgr2=0x7fb55806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.411+0000 7fb556d9d640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580cdf60 0x7fb55806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.411+0000 7fb556d9d640 1 -- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5580cdb80 msgr2=0x7fb5580ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.411+0000 7fb556d9d640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5580cdb80 0x7fb5580ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.411+0000 7fb556d9d640 1 -- 172.21.15.32:0/3784310337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550009a70 con 0x7fb5580c7830 2024-03-16T20:09:26.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.415+0000 7fb556d9d640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c500 tx=0x7fb54800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548005cf0 con 0x7fb5580c7830 2024-03-16T20:09:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548005e50 con 0x7fb5580c7830 2024-03-16T20:09:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5581f2f10 con 0x7fb5580c7830 2024-03-16T20:09:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480072f0 con 0x7fb5580c7830 2024-03-16T20:09:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5581f34d0 con 0x7fb5580c7830 2024-03-16T20:09:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.419+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55806c6b0 con 0x7fb5580c7830 2024-03-16T20:09:26.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.443+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb548007450 con 0x7fb5580c7830 2024-03-16T20:09:26.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.447+0000 7fb53f7fe640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 0x7fb544069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:26.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.447+0000 7fb55659c640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 0x7fb544069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:26.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.447+0000 7fb55659c640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 0x7fb544069300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb5581618c0 tx=0x7fb54c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.751+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb548054e30 con 0x7fb5580c7830 2024-03-16T20:09:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.751+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb5480510d0 con 0x7fb5580c7830 2024-03-16T20:09:26.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:26.995+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb55806c6b0 con 0x7fb5580c7830 2024-03-16T20:09:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.091+0000 7fb53f7fe640 1 -- 172.21.15.32:0/3784310337 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb55806c6b0 con 0x7fb5580c7830 2024-03-16T20:09:27.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:27.085 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 msgr2=0x7fb544069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 0x7fb544069300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb5581618c0 tx=0x7fb54c009190 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 msgr2=0x7fb558154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c500 tx=0x7fb54800c9c0 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 shutdown_connections 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb544066e40 0x7fb544069300 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5580cdf60 0x7fb55806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5580cdb80 0x7fb5580ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 --2- 172.21.15.32:0/3784310337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5580c7830 0x7fb558154b00 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 >> 172.21.15.32:0/3784310337 conn(0x7fb558066ea0 msgr2=0x7fb5580c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 shutdown_connections 2024-03-16T20:09:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:27.095+0000 7fb55d1ad640 1 -- 172.21.15.32:0/3784310337 wait complete. 2024-03-16T20:09:27.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:27.715 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:28.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:28.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 Processor -- start 2024-03-16T20:09:28.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 -- start start 2024-03-16T20:09:28.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f653c15ab30 0x7f653c073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:28.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653c072dd0 con 0x7f653c0725e0 2024-03-16T20:09:28.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653c15d790 con 0x7f653c150d70 2024-03-16T20:09:28.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6542854640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c15da50 con 0x7f653c15ab30 2024-03-16T20:09:28.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6540df1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f653c15ab30 0x7f653c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:28.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6540df1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f653c15ab30 0x7f653c073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48764/0 (socket says 172.21.15.32:48764) 2024-03-16T20:09:28.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.847+0000 7f6540df1640 1 -- 172.21.15.32:0/341254887 learned_addr learned my addr 172.21.15.32:0/341254887 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:28.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.851+0000 7f6540df1640 1 -- 172.21.15.32:0/341254887 >> v1:172.21.15.59:6789/0 conn(0x7f653c150d70 legacy=0x7f653c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:28.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.851+0000 7f6540df1640 1 -- 172.21.15.32:0/341254887 >> v1:172.21.15.32:6789/0 conn(0x7f653c0725e0 legacy=0x7f653c15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:28.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.851+0000 7f6540df1640 1 -- 172.21.15.32:0/341254887 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c15dd30 con 0x7f653c15ab30 2024-03-16T20:09:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.855+0000 7f6540df1640 1 --2- 172.21.15.32:0/341254887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c15ab30 0x7f653c073070 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6534009e50 tx=0x7f653401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c75a3af0cea828 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f653affd640 1 -- 172.21.15.32:0/341254887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340a5760 con 0x7f653c15ab30 2024-03-16T20:09:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f653affd640 1 -- 172.21.15.32:0/341254887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65340a58c0 con 0x7f653c15ab30 2024-03-16T20:09:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f653affd640 1 -- 172.21.15.32:0/341254887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340a5c50 con 0x7f653c15ab30 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 -- 172.21.15.32:0/341254887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c15ab30 msgr2=0x7f653c073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 --2- 172.21.15.32:0/341254887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c15ab30 0x7f653c073070 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6534009e50 tx=0x7f653401c8a0 comp rx=0 tx=0).stop 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 -- 172.21.15.32:0/341254887 shutdown_connections 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 --2- 172.21.15.32:0/341254887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c15ab30 0x7f653c073070 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 -- 172.21.15.32:0/341254887 >> 172.21.15.32:0/341254887 conn(0x7f653c06c0d0 msgr2=0x7f653c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 -- 172.21.15.32:0/341254887 shutdown_connections 2024-03-16T20:09:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.859+0000 7f6542854640 1 -- 172.21.15.32:0/341254887 wait complete. 2024-03-16T20:09:28.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 Processor -- start 2024-03-16T20:09:28.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 -- start start 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f653c0725e0 0x7f653c14c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f653c15ab30 0x7f653c143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c14e060 con 0x7f653c150d70 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c14c020 con 0x7f653c15ab30 2024-03-16T20:09:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6542854640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c14b1c0 con 0x7f653c0725e0 2024-03-16T20:09:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f653b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f6540df1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f653c15ab30 0x7f653c143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f653b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48768/0 (socket says 172.21.15.32:48768) 2024-03-16T20:09:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f653b7fe640 1 -- 172.21.15.32:0/710933089 learned_addr learned my addr 172.21.15.32:0/710933089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.863+0000 7f653bfff640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f653c0725e0 0x7f653c14c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.871+0000 7f653b7fe640 1 -- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f653c0725e0 msgr2=0x7f653c14c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.871+0000 7f653b7fe640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f653c0725e0 0x7f653c14c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.871+0000 7f653b7fe640 1 -- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f653c15ab30 msgr2=0x7f653c143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.871+0000 7f653b7fe640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f653c15ab30 0x7f653c143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.871+0000 7f653b7fe640 1 -- 172.21.15.32:0/710933089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534009b30 con 0x7f653c150d70 2024-03-16T20:09:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.875+0000 7f653b7fe640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f653000b3e0 tx=0x7f653000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.879+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011830 con 0x7f653c150d70 2024-03-16T20:09:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.879+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65300119b0 con 0x7f653c150d70 2024-03-16T20:09:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.879+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011d60 con 0x7f653c150d70 2024-03-16T20:09:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.879+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c15d560 con 0x7f653c150d70 2024-03-16T20:09:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.879+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c14b780 con 0x7f653c150d70 2024-03-16T20:09:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:28.883+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c068000 con 0x7f653c150d70 2024-03-16T20:09:29.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.159+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f653001a400 con 0x7f653c150d70 2024-03-16T20:09:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.159+0000 7f6538ff9640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 0x7f6528069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.159+0000 7f653bfff640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 0x7f6528069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.159+0000 7f653bfff640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 0x7f6528069380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f652c001770 tx=0x7f652c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:29.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.187+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6530055c70 con 0x7f653c150d70 2024-03-16T20:09:29.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.187+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6530051f10 con 0x7f653c150d70 2024-03-16T20:09:29.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.443+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f653c0722b0 con 0x7f653c150d70 2024-03-16T20:09:29.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.459+0000 7f6538ff9640 1 -- 172.21.15.32:0/710933089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6530015090 con 0x7f653c150d70 2024-03-16T20:09:29.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:29.451 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:29.451 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:29.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.459+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 msgr2=0x7f6528069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.459+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 0x7f6528069380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f652c001770 tx=0x7f652c009040 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.459+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 msgr2=0x7f653c146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.459+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f653000b3e0 tx=0x7f653000b8a0 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 shutdown_connections 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6528066ec0 0x7f6528069380 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f653c15ab30 0x7f653c143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f653c150d70 0x7f653c146520 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 --2- 172.21.15.32:0/710933089 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f653c0725e0 0x7f653c14c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 >> 172.21.15.32:0/710933089 conn(0x7f653c06c0d0 msgr2=0x7f653c0c8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 shutdown_connections 2024-03-16T20:09:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:29.463+0000 7f6542854640 1 -- 172.21.15.32:0/710933089 wait complete. 2024-03-16T20:09:30.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:30.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:31.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 Processor -- start 2024-03-16T20:09:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 -- start start 2024-03-16T20:09:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1240cdf60 0x7fe124151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe124157eb0 con 0x7fe1240c7830 2024-03-16T20:09:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1240c4c80 con 0x7fe1240cdb80 2024-03-16T20:09:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.190+0000 7fe12b948640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12406fa80 con 0x7fe1240cdf60 2024-03-16T20:09:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.194+0000 7fe128ee3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1240cdf60 0x7fe124151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.194+0000 7fe1296e4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe1240cdb80 0x7fe12406f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:42598/0 (socket says 172.21.15.32:42598) 2024-03-16T20:09:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.194+0000 7fe1296e4640 1 -- 172.21.15.32:0/1801457997 learned_addr learned my addr 172.21.15.32:0/1801457997 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.198+0000 7fe128ee3640 1 -- 172.21.15.32:0/1801457997 >> v1:172.21.15.59:6789/0 conn(0x7fe1240cdb80 legacy=0x7fe12406f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.198+0000 7fe128ee3640 1 -- 172.21.15.32:0/1801457997 >> v1:172.21.15.32:6789/0 conn(0x7fe1240c7830 legacy=0x7fe124158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.198+0000 7fe128ee3640 1 -- 172.21.15.32:0/1801457997 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1240c4f40 con 0x7fe1240cdf60 2024-03-16T20:09:31.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.202+0000 7fe128ee3640 1 --2- 172.21.15.32:0/1801457997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdf60 0x7fe124151b20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe1140097c0 tx=0x7fe11401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae5f50192dc5a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe11bfff640 1 -- 172.21.15.32:0/1801457997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140a7070 con 0x7fe1240cdf60 2024-03-16T20:09:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe11bfff640 1 -- 172.21.15.32:0/1801457997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140ab430 con 0x7fe1240cdf60 2024-03-16T20:09:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe11bfff640 1 -- 172.21.15.32:0/1801457997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140ab7c0 con 0x7fe1240cdf60 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 -- 172.21.15.32:0/1801457997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdf60 msgr2=0x7fe124151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 --2- 172.21.15.32:0/1801457997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdf60 0x7fe124151b20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe1140097c0 tx=0x7fe11401c430 comp rx=0 tx=0).stop 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 -- 172.21.15.32:0/1801457997 shutdown_connections 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 --2- 172.21.15.32:0/1801457997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdf60 0x7fe124151b20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 -- 172.21.15.32:0/1801457997 >> 172.21.15.32:0/1801457997 conn(0x7fe124066ea0 msgr2=0x7fe1240672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 -- 172.21.15.32:0/1801457997 shutdown_connections 2024-03-16T20:09:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.206+0000 7fe12b948640 1 -- 172.21.15.32:0/1801457997 wait complete. 2024-03-16T20:09:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 Processor -- start 2024-03-16T20:09:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 -- start start 2024-03-16T20:09:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1240c7830 0x7fe124156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1240cdf60 0x7fe1241f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12414a700 con 0x7fe1240cdb80 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe124157490 con 0x7fe1240cdf60 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe12b948640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12414af80 con 0x7fe1240c7830 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe128ee3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.210+0000 7fe128ee3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41108/0 (socket says 172.21.15.32:41108) 2024-03-16T20:09:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.214+0000 7fe128ee3640 1 -- 172.21.15.32:0/4197830061 learned_addr learned my addr 172.21.15.32:0/4197830061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.214+0000 7fe129ee5640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1240cdf60 0x7fe1241f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.214+0000 7fe1296e4640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1240c7830 0x7fe124156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 -- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1240c7830 msgr2=0x7fe124156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1240c7830 0x7fe124156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 -- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1240cdf60 msgr2=0x7fe1241f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1240cdf60 0x7fe1241f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 -- 172.21.15.32:0/4197830061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009420 con 0x7fe1240cdb80 2024-03-16T20:09:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.218+0000 7fe128ee3640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe114019040 tx=0x7fe11401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.222+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140b5810 con 0x7fe1240cdb80 2024-03-16T20:09:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.222+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140b5990 con 0x7fe1240cdb80 2024-03-16T20:09:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.222+0000 7fe12b948640 1 -- 172.21.15.32:0/4197830061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1240ccfd0 con 0x7fe1240cdb80 2024-03-16T20:09:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.222+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140b5d40 con 0x7fe1240cdb80 2024-03-16T20:09:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.222+0000 7fe12b948640 1 -- 172.21.15.32:0/4197830061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12414a930 con 0x7fe1240cdb80 2024-03-16T20:09:31.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.226+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc0052a0 con 0x7fe1240cdb80 2024-03-16T20:09:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.506+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe1140c8420 con 0x7fe1240cdb80 2024-03-16T20:09:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.506+0000 7fe119ffb640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 0x7fe108069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.506+0000 7fe1296e4640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 0x7fe108069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.506+0000 7fe1296e4640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 0x7fe108069300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe120000f70 tx=0x7fe120007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.522+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe1140f0240 con 0x7fe1240cdb80 2024-03-16T20:09:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.522+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe1140ec4e0 con 0x7fe1240cdb80 2024-03-16T20:09:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.742+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0fc005900 con 0x7fe1240cdb80 2024-03-16T20:09:31.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.782+0000 7fe119ffb640 1 -- 172.21.15.32:0/4197830061 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fe0fc005900 con 0x7fe1240cdb80 2024-03-16T20:09:31.776 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:31.776 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:31.776 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:31.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 msgr2=0x7fe108069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 0x7fe108069300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe120000f70 tx=0x7fe120007e90 comp rx=0 tx=0).stop 2024-03-16T20:09:31.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 msgr2=0x7fe124148a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe114019040 tx=0x7fe11401ac70 comp rx=0 tx=0).stop 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 shutdown_connections 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe108066e40 0x7fe108069300 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1240cdf60 0x7fe1241f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1240cdb80 0x7fe124148a80 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 --2- 172.21.15.32:0/4197830061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe1240c7830 0x7fe124156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 >> 172.21.15.32:0/4197830061 conn(0x7fe124066ea0 msgr2=0x7fe1240c8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 shutdown_connections 2024-03-16T20:09:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:31.786+0000 7fe10f7fe640 1 -- 172.21.15.32:0/4197830061 wait complete. 2024-03-16T20:09:32.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:32.405 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:33.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.522+0000 7f8482a5c640 1 Processor -- start 2024-03-16T20:09:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.522+0000 7f8482a5c640 1 -- start start 2024-03-16T20:09:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.522+0000 7f8482a5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f847c073120 0x7f847c157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8482a5c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8482a5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c150fa0 con 0x7f847c150bc0 2024-03-16T20:09:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8482a5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c0c7c30 con 0x7f847c073120 2024-03-16T20:09:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8482a5c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c071b00 con 0x7f847c0c69e0 2024-03-16T20:09:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f847b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f847c073120 0x7f847c157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8480ff9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8480ff9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:37580/0 (socket says 172.21.15.32:37580) 2024-03-16T20:09:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.526+0000 7f8480ff9640 1 -- 172.21.15.32:0/3766879454 learned_addr learned my addr 172.21.15.32:0/3766879454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.530+0000 7f8480ff9640 1 -- 172.21.15.32:0/3766879454 >> v2:172.21.15.59:3300/0 conn(0x7f847c073120 msgr2=0x7f847c157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.530+0000 7f8480ff9640 1 --2- 172.21.15.32:0/3766879454 >> v2:172.21.15.59:3300/0 conn(0x7f847c073120 0x7f847c157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.530+0000 7f8480ff9640 1 -- 172.21.15.32:0/3766879454 >> v1:172.21.15.59:6789/0 conn(0x7f847c150bc0 legacy=0x7f847c0c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.530+0000 7f8480ff9640 1 -- 172.21.15.32:0/3766879454 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c071da0 con 0x7f847c0c69e0 2024-03-16T20:09:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.534+0000 7f8480ff9640 1 --2- 172.21.15.32:0/3766879454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8474009e50 tx=0x7f847401f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f0bdb26aa93c6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f847affd640 1 -- 172.21.15.32:0/3766879454 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84740af400 con 0x7f847c0c69e0 2024-03-16T20:09:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f847affd640 1 -- 172.21.15.32:0/3766879454 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84740ab050 con 0x7f847c0c69e0 2024-03-16T20:09:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f847affd640 1 -- 172.21.15.32:0/3766879454 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84740af780 con 0x7f847c0c69e0 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- 172.21.15.32:0/3766879454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 msgr2=0x7f847c0c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3766879454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8474009e50 tx=0x7f847401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- 172.21.15.32:0/3766879454 shutdown_connections 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3766879454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c0c6e20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3766879454 >> v2:172.21.15.59:3300/0 conn(0x7f847c073120 0x7f847c157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- 172.21.15.32:0/3766879454 >> 172.21.15.32:0/3766879454 conn(0x7f847c0cbfe0 msgr2=0x7f847c0cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- 172.21.15.32:0/3766879454 shutdown_connections 2024-03-16T20:09:33.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- 172.21.15.32:0/3766879454 wait complete. 2024-03-16T20:09:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 Processor -- start 2024-03-16T20:09:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.542+0000 7f8482a5c640 1 -- start start 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f847c150bc0 0x7f847c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c14d170 con 0x7f847c073120 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c147ce0 con 0x7f847c150bc0 2024-03-16T20:09:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8482a5c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1482a0 con 0x7f847c0c69e0 2024-03-16T20:09:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f847bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f8480ff9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f847c150bc0 0x7f847c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f847bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41120/0 (socket says 172.21.15.32:41120) 2024-03-16T20:09:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c148730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:37596/0 (socket says 172.21.15.32:37596) 2024-03-16T20:09:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.546+0000 7f847bfff640 1 -- 172.21.15.32:0/3097223290 learned_addr learned my addr 172.21.15.32:0/3097223290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.550+0000 7f847bfff640 1 -- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 msgr2=0x7f847c148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.550+0000 7f847bfff640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.550+0000 7f847bfff640 1 -- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f847c150bc0 msgr2=0x7f847c148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.550+0000 7f847bfff640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f847c150bc0 0x7f847c148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.550+0000 7f847bfff640 1 -- 172.21.15.32:0/3097223290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474009b30 con 0x7f847c073120 2024-03-16T20:09:33.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.554+0000 7f847bfff640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d9c0 tx=0x7f846c00de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00ed40 con 0x7f847c073120 2024-03-16T20:09:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c016440 con 0x7f847c073120 2024-03-16T20:09:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c14d3a0 con 0x7f847c073120 2024-03-16T20:09:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00eea0 con 0x7f847c073120 2024-03-16T20:09:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c147f30 con 0x7f847c073120 2024-03-16T20:09:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.562+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c15a440 con 0x7f847c073120 2024-03-16T20:09:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.854+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f846c01f3b0 con 0x7f847c073120 2024-03-16T20:09:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.854+0000 7f8478ff9640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 0x7f8468069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.854+0000 7f847b7fe640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 0x7f8468069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.858+0000 7f847b7fe640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 0x7f8468069350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f847c0cc8b0 tx=0x7f8470009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:33.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.882+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f846c0579f0 con 0x7f847c073120 2024-03-16T20:09:33.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:33.882+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f846c053c90 con 0x7f847c073120 2024-03-16T20:09:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.206+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f847c1480c0 con 0x7f847c073120 2024-03-16T20:09:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.226+0000 7f8478ff9640 1 -- 172.21.15.32:0/3097223290 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f846c01b020 con 0x7f847c073120 2024-03-16T20:09:34.219 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:34.220 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 msgr2=0x7f8468069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 0x7f8468069350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f847c0cc8b0 tx=0x7f8470009190 comp rx=0 tx=0).stop 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 msgr2=0x7f847c1efc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d9c0 tx=0x7f846c00de80 comp rx=0 tx=0).stop 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 shutdown_connections 2024-03-16T20:09:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8468066e90 0x7f8468069350 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f847c150bc0 0x7f847c148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f847c0c69e0 0x7f847c148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 --2- 172.21.15.32:0/3097223290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f847c073120 0x7f847c1efc80 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 >> 172.21.15.32:0/3097223290 conn(0x7f847c0cbfe0 msgr2=0x7f847c0cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 shutdown_connections 2024-03-16T20:09:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:34.230+0000 7f8482a5c640 1 -- 172.21.15.32:0/3097223290 wait complete. 2024-03-16T20:09:34.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:34.843 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:35.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 Processor -- start 2024-03-16T20:09:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 -- start start 2024-03-16T20:09:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f14fc151ef0 0x7f14fc0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c97b0 0x7f14fc0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc158c80 con 0x7f14fc0c8720 2024-03-16T20:09:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14fc0b0ad0 con 0x7f14fc151ef0 2024-03-16T20:09:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f15008b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc157cb0 con 0x7f14fc0c97b0 2024-03-16T20:09:35.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f14fad9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f14fc0c8720 0x7f14fc15b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47550/0 (socket says 172.21.15.32:47550) 2024-03-16T20:09:35.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f14fad9d640 1 -- 172.21.15.32:0/3068437029 learned_addr learned my addr 172.21.15.32:0/3068437029 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:35.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f14f9d9b640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c97b0 0x7f14fc0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:35.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.962+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3068437029 >> v2:172.21.15.59:3300/0 conn(0x7f14fc151ef0 0x7f14fc0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:35.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.966+0000 7f14f959a640 1 -- 172.21.15.32:0/3068437029 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181850292 0 0) 0x7f14fc158c80 con 0x7f14fc0c8720 2024-03-16T20:09:35.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.966+0000 7f14f959a640 1 -- 172.21.15.32:0/3068437029 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e4003530 con 0x7f14fc0c8720 2024-03-16T20:09:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.970+0000 7f14fa59c640 1 -- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c97b0 msgr2=0x7f14fc0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.970+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c97b0 0x7f14fc0c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.970+0000 7f14fa59c640 1 -- 172.21.15.32:0/3068437029 >> v1:172.21.15.32:6789/0 conn(0x7f14fc0c8720 legacy=0x7f14fc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.970+0000 7f14fa59c640 1 -- 172.21.15.32:0/3068437029 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc15bfc0 con 0x7f14fc151ef0 2024-03-16T20:09:35.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.978+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc0c9270 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097c0 tx=0x7f14ec01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7376d397aea16ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:35.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f14f959a640 1 -- 172.21.15.32:0/3068437029 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0a8990 con 0x7f14fc151ef0 2024-03-16T20:09:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f14f959a640 1 -- 172.21.15.32:0/3068437029 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0a8af0 con 0x7f14fc151ef0 2024-03-16T20:09:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f14f959a640 1 -- 172.21.15.32:0/3068437029 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0a8e80 con 0x7f14fc151ef0 2024-03-16T20:09:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 -- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 msgr2=0x7f14fc0c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc0c9270 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097c0 tx=0x7f14ec01ec90 comp rx=0 tx=0).stop 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 -- 172.21.15.32:0/3068437029 shutdown_connections 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c97b0 0x7f14fc0c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 --2- 172.21.15.32:0/3068437029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc0c9270 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 -- 172.21.15.32:0/3068437029 >> 172.21.15.32:0/3068437029 conn(0x7f14fc070f90 msgr2=0x7f14fc0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 -- 172.21.15.32:0/3068437029 shutdown_connections 2024-03-16T20:09:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.986+0000 7f15008b6640 1 -- 172.21.15.32:0/3068437029 wait complete. 2024-03-16T20:09:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 Processor -- start 2024-03-16T20:09:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 -- start start 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c8720 0x7f14fc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc14add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc06d970 con 0x7f14fc0c97b0 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc1efbc0 con 0x7f14fc151ef0 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f15008b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc1f0b90 con 0x7f14fc0c8720 2024-03-16T20:09:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f14f9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f14f9d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41124/0 (socket says 172.21.15.32:41124) 2024-03-16T20:09:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f14f9d9b640 1 -- 172.21.15.32:0/3481193345 learned_addr learned my addr 172.21.15.32:0/3481193345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c8720 0x7f14fc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.990+0000 7f14fad9d640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc14add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:35.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.998+0000 7f14f9d9b640 1 -- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c8720 msgr2=0x7f14fc0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:35.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.998+0000 7f14f9d9b640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c8720 0x7f14fc0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:35.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.998+0000 7f14f9d9b640 1 -- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 msgr2=0x7f14fc14add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:35.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.998+0000 7f14f9d9b640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc14add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:35.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:35.998+0000 7f14f9d9b640 1 -- 172.21.15.32:0/3481193345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec009420 con 0x7f14fc0c97b0 2024-03-16T20:09:35.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.002+0000 7f14f9d9b640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f14f000e900 tx=0x7f14f000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0009690 con 0x7f14fc0c97b0 2024-03-16T20:09:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f000ca00 con 0x7f14fc0c97b0 2024-03-16T20:09:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc1f0700 con 0x7f14fc0c97b0 2024-03-16T20:09:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f000cdb0 con 0x7f14fc0c97b0 2024-03-16T20:09:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc06d690 con 0x7f14fc0c97b0 2024-03-16T20:09:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.006+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc1f0890 con 0x7f14fc0c97b0 2024-03-16T20:09:36.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.038+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14f0013470 con 0x7f14fc0c97b0 2024-03-16T20:09:36.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.038+0000 7f14eaffd640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 0x7f14dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:36.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.038+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 0x7f14dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:36.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.038+0000 7f14fa59c640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 0x7f14dc069300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b2000 tx=0x7f14ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:36.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.318+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f14f0055bd0 con 0x7f14fc0c97b0 2024-03-16T20:09:36.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.318+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f14f0051e70 con 0x7f14fc0c97b0 2024-03-16T20:09:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.526+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14fc0b3b60 con 0x7f14fc0c97b0 2024-03-16T20:09:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.542+0000 7f14eaffd640 1 -- 172.21.15.32:0/3481193345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f14f0021070 con 0x7f14fc0c97b0 2024-03-16T20:09:36.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:36.534 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 msgr2=0x7f14dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 0x7f14dc069300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b2000 tx=0x7f14ec008040 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 msgr2=0x7f14fc06c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f14f000e900 tx=0x7f14f000edc0 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 shutdown_connections 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f14dc066e40 0x7f14dc069300 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14fc151ef0 0x7f14fc14add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14fc0c97b0 0x7f14fc06c0d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 --2- 172.21.15.32:0/3481193345 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14fc0c8720 0x7f14fc0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 >> 172.21.15.32:0/3481193345 conn(0x7f14fc070f90 msgr2=0x7f14fc155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 shutdown_connections 2024-03-16T20:09:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:36.546+0000 7f15008b6640 1 -- 172.21.15.32:0/3481193345 wait complete. 2024-03-16T20:09:37.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:37.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:38.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:38.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 Processor -- start 2024-03-16T20:09:38.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 -- start start 2024-03-16T20:09:38.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 0x7faacc073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faacc072dd0 con 0x7faacc0725e0 2024-03-16T20:09:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faacc15d790 con 0x7faacc15a420 2024-03-16T20:09:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faad1307640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faacc15da50 con 0x7faacc150d70 2024-03-16T20:09:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faaca7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faacc0725e0 0x7faacc070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47564/0 (socket says 172.21.15.32:47564) 2024-03-16T20:09:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.278+0000 7faaca7fc640 1 -- 172.21.15.32:0/3299357392 learned_addr learned my addr 172.21.15.32:0/3299357392 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:38.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.282+0000 7faacaffd640 1 --2- 172.21.15.32:0/3299357392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 0x7faacc073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:38.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.282+0000 7faac9ffb640 1 -- 172.21.15.32:0/3299357392 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566818368 0 0) 0x7faacc072dd0 con 0x7faacc0725e0 2024-03-16T20:09:38.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.282+0000 7faac9ffb640 1 -- 172.21.15.32:0/3299357392 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab4003530 con 0x7faacc0725e0 2024-03-16T20:09:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.286+0000 7faacaffd640 1 -- 172.21.15.32:0/3299357392 >> v1:172.21.15.59:6789/0 conn(0x7faacc15a420 legacy=0x7faacc15a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.286+0000 7faacaffd640 1 -- 172.21.15.32:0/3299357392 >> v1:172.21.15.32:6789/0 conn(0x7faacc0725e0 legacy=0x7faacc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.286+0000 7faacaffd640 1 -- 172.21.15.32:0/3299357392 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc15dd30 con 0x7faacc150d70 2024-03-16T20:09:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.290+0000 7faacaffd640 1 --2- 172.21.15.32:0/3299357392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 0x7faacc073070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faabc0097f0 tx=0x7faabc01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=187da44fdf66c9e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:38.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faac9ffb640 1 -- 172.21.15.32:0/3299357392 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc0ab620 con 0x7faacc150d70 2024-03-16T20:09:38.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faac9ffb640 1 -- 172.21.15.32:0/3299357392 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc0ab780 con 0x7faacc150d70 2024-03-16T20:09:38.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faac9ffb640 1 -- 172.21.15.32:0/3299357392 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc0abb10 con 0x7faacc150d70 2024-03-16T20:09:38.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- 172.21.15.32:0/3299357392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 msgr2=0x7faacc073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 --2- 172.21.15.32:0/3299357392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 0x7faacc073070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faabc0097f0 tx=0x7faabc01c420 comp rx=0 tx=0).stop 2024-03-16T20:09:38.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- 172.21.15.32:0/3299357392 shutdown_connections 2024-03-16T20:09:38.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 --2- 172.21.15.32:0/3299357392 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc150d70 0x7faacc073070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- 172.21.15.32:0/3299357392 >> 172.21.15.32:0/3299357392 conn(0x7faacc06c0d0 msgr2=0x7faacc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:38.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- 172.21.15.32:0/3299357392 shutdown_connections 2024-03-16T20:09:38.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- 172.21.15.32:0/3299357392 wait complete. 2024-03-16T20:09:38.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 Processor -- start 2024-03-16T20:09:38.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.298+0000 7faad1307640 1 -- start start 2024-03-16T20:09:38.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc0725e0 0x7faacc0cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:38.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faacc150d70 0x7faacc14fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:38.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:38.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc14ffc0 con 0x7faacc15a420 2024-03-16T20:09:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc1414e0 con 0x7faacc150d70 2024-03-16T20:09:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faad1307640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faacc15cd40 con 0x7faacc0725e0 2024-03-16T20:09:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faacb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faacaffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc0725e0 0x7faacc0cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faaca7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faacc150d70 0x7faacc14fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faacb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41134/0 (socket says 172.21.15.32:41134) 2024-03-16T20:09:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.302+0000 7faacb7fe640 1 -- 172.21.15.32:0/2103999067 learned_addr learned my addr 172.21.15.32:0/2103999067 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.306+0000 7faacb7fe640 1 -- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc0725e0 msgr2=0x7faacc0cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.306+0000 7faacb7fe640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc0725e0 0x7faacc0cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.306+0000 7faacb7fe640 1 -- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faacc150d70 msgr2=0x7faacc14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.306+0000 7faacb7fe640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faacc150d70 0x7faacc14fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.306+0000 7faacb7fe640 1 -- 172.21.15.32:0/2103999067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc009420 con 0x7faacc15a420 2024-03-16T20:09:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.310+0000 7faacb7fe640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faac400c730 tx=0x7faac400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:38.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4003b80 con 0x7faacc15a420 2024-03-16T20:09:38.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4003ce0 con 0x7faacc15a420 2024-03-16T20:09:38.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faacc15d560 con 0x7faacc15a420 2024-03-16T20:09:38.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac400aa30 con 0x7faacc15a420 2024-03-16T20:09:38.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faacc15cfe0 con 0x7faacc15a420 2024-03-16T20:09:38.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.314+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc068000 con 0x7faacc15a420 2024-03-16T20:09:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.654+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faac400ab90 con 0x7faacc15a420 2024-03-16T20:09:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.654+0000 7faab37fe640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 0x7faab8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.654+0000 7faacaffd640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 0x7faab8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.654+0000 7faacaffd640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 0x7faab8069330 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faabc0af000 tx=0x7faabc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.682+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7faac400ae40 con 0x7faacc15a420 2024-03-16T20:09:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.682+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7faac4052040 con 0x7faacc15a420 2024-03-16T20:09:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:38.906+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faacc141620 con 0x7faacc15a420 2024-03-16T20:09:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.002+0000 7faab37fe640 1 -- 172.21.15.32:0/2103999067 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7faac4055700 con 0x7faacc15a420 2024-03-16T20:09:38.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:38.995 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 msgr2=0x7faab8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 0x7faab8069330 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faabc0af000 tx=0x7faabc0a7000 comp rx=0 tx=0).stop 2024-03-16T20:09:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 msgr2=0x7faacc142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faac400c730 tx=0x7faac400cbf0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 shutdown_connections 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7faab8066e70 0x7faab8069330 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc15a420 0x7faacc142f80 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faacc150d70 0x7faacc14fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 --2- 172.21.15.32:0/2103999067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faacc0725e0 0x7faacc0cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 >> 172.21.15.32:0/2103999067 conn(0x7faacc06c0d0 msgr2=0x7faacc0c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 shutdown_connections 2024-03-16T20:09:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:39.006+0000 7faad1307640 1 -- 172.21.15.32:0/2103999067 wait complete. 2024-03-16T20:09:39.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:39.621 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:40.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 Processor -- start 2024-03-16T20:09:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 -- start start 2024-03-16T20:09:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f025c151ef0 0x7f025c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f025c15b2a0 0x7f025c15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c158c80 con 0x7f025c0c97b0 2024-03-16T20:09:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f025c0b0ad0 con 0x7f025c151ef0 2024-03-16T20:09:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f02635df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f025c157cb0 con 0x7f025c15b2a0 2024-03-16T20:09:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f026137b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f025c151ef0 0x7f025c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f0261b7c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f025c15b2a0 0x7f025c15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f026137b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f025c151ef0 0x7f025c0c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58758/0 (socket says 172.21.15.32:58758) 2024-03-16T20:09:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.742+0000 7f026137b640 1 -- 172.21.15.32:0/88265796 learned_addr learned my addr 172.21.15.32:0/88265796 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:40.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.746+0000 7f026137b640 1 -- 172.21.15.32:0/88265796 >> v2:172.21.15.59:3300/0 conn(0x7f025c15b2a0 msgr2=0x7f025c15b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:40.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.746+0000 7f026137b640 1 --2- 172.21.15.32:0/88265796 >> v2:172.21.15.59:3300/0 conn(0x7f025c15b2a0 0x7f025c15b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:40.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.746+0000 7f026137b640 1 -- 172.21.15.32:0/88265796 >> v1:172.21.15.59:6789/0 conn(0x7f025c0c97b0 legacy=0x7f025c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:40.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.746+0000 7f026137b640 1 -- 172.21.15.32:0/88265796 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c15bfc0 con 0x7f025c151ef0 2024-03-16T20:09:40.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.750+0000 7f026137b640 1 --2- 172.21.15.32:0/88265796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9270 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f025c06ff60 tx=0x7f025801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bb3886040c31bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.754+0000 7f0253fff640 1 -- 172.21.15.32:0/88265796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02580a8a10 con 0x7f025c151ef0 2024-03-16T20:09:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.754+0000 7f0253fff640 1 -- 172.21.15.32:0/88265796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02580a8b70 con 0x7f025c151ef0 2024-03-16T20:09:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.754+0000 7f0253fff640 1 -- 172.21.15.32:0/88265796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02580b0440 con 0x7f025c151ef0 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- 172.21.15.32:0/88265796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 msgr2=0x7f025c0c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 --2- 172.21.15.32:0/88265796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9270 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f025c06ff60 tx=0x7f025801ed10 comp rx=0 tx=0).stop 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- 172.21.15.32:0/88265796 shutdown_connections 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 --2- 172.21.15.32:0/88265796 >> v2:172.21.15.59:3300/0 conn(0x7f025c15b2a0 0x7f025c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 --2- 172.21.15.32:0/88265796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9270 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- 172.21.15.32:0/88265796 >> 172.21.15.32:0/88265796 conn(0x7f025c070f90 msgr2=0x7f025c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- 172.21.15.32:0/88265796 shutdown_connections 2024-03-16T20:09:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- 172.21.15.32:0/88265796 wait complete. 2024-03-16T20:09:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 Processor -- start 2024-03-16T20:09:40.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.758+0000 7f02635df640 1 -- start start 2024-03-16T20:09:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f025c0c97b0 0x7f025c1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f025c15b2a0 0x7f025c0b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c1f4380 con 0x7f025c151ef0 2024-03-16T20:09:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c1f0b40 con 0x7f025c15b2a0 2024-03-16T20:09:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f02635df640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c06c510 con 0x7f025c0c97b0 2024-03-16T20:09:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f0260b7a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f0260b7a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58770/0 (socket says 172.21.15.32:58770) 2024-03-16T20:09:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f0260b7a640 1 -- 172.21.15.32:0/3156319398 learned_addr learned my addr 172.21.15.32:0/3156319398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f026137b640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f025c0c97b0 0x7f025c1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.762+0000 7f0261b7c640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f025c15b2a0 0x7f025c0b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.766+0000 7f0260b7a640 1 -- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f025c0c97b0 msgr2=0x7f025c1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:40.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.766+0000 7f0260b7a640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f025c0c97b0 0x7f025c1f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:40.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.766+0000 7f0260b7a640 1 -- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f025c15b2a0 msgr2=0x7f025c0b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:40.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.766+0000 7f0260b7a640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f025c15b2a0 0x7f025c0b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:40.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.766+0000 7f0260b7a640 1 -- 172.21.15.32:0/3156319398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580094d0 con 0x7f025c151ef0 2024-03-16T20:09:40.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.770+0000 7f0260b7a640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f024c00c430 tx=0x7f024c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:40.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c0052d0 con 0x7f025c151ef0 2024-03-16T20:09:40.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c005430 con 0x7f025c151ef0 2024-03-16T20:09:40.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c007260 con 0x7f025c151ef0 2024-03-16T20:09:40.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025c1f0210 con 0x7f025c151ef0 2024-03-16T20:09:40.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025c1f07d0 con 0x7f025c151ef0 2024-03-16T20:09:40.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.774+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025c1f0990 con 0x7f025c151ef0 2024-03-16T20:09:40.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.802+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f024c0073c0 con 0x7f025c151ef0 2024-03-16T20:09:40.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.802+0000 7f0251ffb640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 0x7f0240069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:40.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.802+0000 7f026137b640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 0x7f0240069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:40.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.802+0000 7f026137b640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 0x7f0240069300 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f02580b2000 tx=0x7f02580aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:40.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:40.802+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f024c007670 con 0x7f025c151ef0 2024-03-16T20:09:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.078+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f024c051170 con 0x7f025c151ef0 2024-03-16T20:09:41.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.286+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f025c068000 con 0x7f025c151ef0 2024-03-16T20:09:41.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.310+0000 7f0251ffb640 1 -- 172.21.15.32:0/3156319398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f024c020070 con 0x7f025c151ef0 2024-03-16T20:09:41.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:41.305 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:41.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:41.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 msgr2=0x7f0240069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:41.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 0x7f0240069300 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f02580b2000 tx=0x7f02580aa070 comp rx=0 tx=0).stop 2024-03-16T20:09:41.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 msgr2=0x7f025c0c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:41.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f024c00c430 tx=0x7f024c00c8f0 comp rx=0 tx=0).stop 2024-03-16T20:09:41.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 shutdown_connections 2024-03-16T20:09:41.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0240066e40 0x7f0240069300 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:41.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f025c15b2a0 0x7f025c0b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:41.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f025c151ef0 0x7f025c0c9e40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:41.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 --2- 172.21.15.32:0/3156319398 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f025c0c97b0 0x7f025c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:41.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 >> 172.21.15.32:0/3156319398 conn(0x7f025c070f90 msgr2=0x7f025c0c36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:41.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 shutdown_connections 2024-03-16T20:09:41.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:41.314+0000 7f02635df640 1 -- 172.21.15.32:0/3156319398 wait complete. 2024-03-16T20:09:41.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:41.929 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:42.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:09:42.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:09:42.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 Processor -- start 2024-03-16T20:09:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 -- start start 2024-03-16T20:09:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88b81543f0 0x7f88b815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88b80c8bb0 0x7f88b806f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b815d5a0 con 0x7f88b815e250 2024-03-16T20:09:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b815d880 con 0x7f88b81543f0 2024-03-16T20:09:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.062+0000 7f88bd1b1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b806ba40 con 0x7f88b80c8bb0 2024-03-16T20:09:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.066+0000 7f88b6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88b81543f0 0x7f88b815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.066+0000 7f88b659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88b80c8bb0 0x7f88b806f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.066+0000 7f88b6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88b81543f0 0x7f88b815dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58780/0 (socket says 172.21.15.32:58780) 2024-03-16T20:09:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.066+0000 7f88b6d9d640 1 -- 172.21.15.32:0/53942888 learned_addr learned my addr 172.21.15.32:0/53942888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.070+0000 7f88b6d9d640 1 -- 172.21.15.32:0/53942888 >> v2:172.21.15.59:3300/0 conn(0x7f88b80c8bb0 msgr2=0x7f88b806f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.070+0000 7f88b6d9d640 1 --2- 172.21.15.32:0/53942888 >> v2:172.21.15.59:3300/0 conn(0x7f88b80c8bb0 0x7f88b806f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.070+0000 7f88b6d9d640 1 -- 172.21.15.32:0/53942888 >> v1:172.21.15.59:6789/0 conn(0x7f88b815e250 legacy=0x7f88b8072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.070+0000 7f88b6d9d640 1 -- 172.21.15.32:0/53942888 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b815ae70 con 0x7f88b81543f0 2024-03-16T20:09:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.074+0000 7f88b6d9d640 1 --2- 172.21.15.32:0/53942888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81543f0 0x7f88b815dd10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f88b80737a0 tx=0x7f88a801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d9fc7c45377021 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88b5d9b640 1 -- 172.21.15.32:0/53942888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a80a89d0 con 0x7f88b81543f0 2024-03-16T20:09:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88b5d9b640 1 -- 172.21.15.32:0/53942888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a80a8b30 con 0x7f88b81543f0 2024-03-16T20:09:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88b5d9b640 1 -- 172.21.15.32:0/53942888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a80b0440 con 0x7f88b81543f0 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 -- 172.21.15.32:0/53942888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81543f0 msgr2=0x7f88b815dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 --2- 172.21.15.32:0/53942888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81543f0 0x7f88b815dd10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f88b80737a0 tx=0x7f88a801ecd0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 -- 172.21.15.32:0/53942888 shutdown_connections 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 --2- 172.21.15.32:0/53942888 >> v2:172.21.15.59:3300/0 conn(0x7f88b80c8bb0 0x7f88b806f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 --2- 172.21.15.32:0/53942888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81543f0 0x7f88b815dd10 secure :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f88b80737a0 tx=0x7f88a801ecd0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 -- 172.21.15.32:0/53942888 >> 172.21.15.32:0/53942888 conn(0x7f88b80cc7a0 msgr2=0x7f88b80ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 -- 172.21.15.32:0/53942888 shutdown_connections 2024-03-16T20:09:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.078+0000 7f88bd1b1640 1 -- 172.21.15.32:0/53942888 wait complete. 2024-03-16T20:09:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 Processor -- start 2024-03-16T20:09:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 -- start start 2024-03-16T20:09:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88b80c8bb0 0x7f88b81505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b815e250 0x7f88b815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b81f1430 con 0x7f88b81e6200 2024-03-16T20:09:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b81e4990 con 0x7f88b815e250 2024-03-16T20:09:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88bd1b1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c4ea0 con 0x7f88b80c8bb0 2024-03-16T20:09:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88b759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88b659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b815e250 0x7f88b815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88b759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58790/0 (socket says 172.21.15.32:58790) 2024-03-16T20:09:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88b759e640 1 -- 172.21.15.32:0/1527542430 learned_addr learned my addr 172.21.15.32:0/1527542430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.082+0000 7f88b6d9d640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88b80c8bb0 0x7f88b81505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.090+0000 7f88b759e640 1 -- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88b80c8bb0 msgr2=0x7f88b81505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.090+0000 7f88b759e640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88b80c8bb0 0x7f88b81505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.090+0000 7f88b759e640 1 -- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b815e250 msgr2=0x7f88b815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.090+0000 7f88b759e640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b815e250 0x7f88b815cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.090+0000 7f88b759e640 1 -- 172.21.15.32:0/1527542430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80094d0 con 0x7f88b81e6200 2024-03-16T20:09:43.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.094+0000 7f88b759e640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c780 tx=0x7f88b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000aa50 con 0x7f88b81e6200 2024-03-16T20:09:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b000abf0 con 0x7f88b81e6200 2024-03-16T20:09:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000ce30 con 0x7f88b81e6200 2024-03-16T20:09:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f88bd1b1640 1 -- 172.21.15.32:0/1527542430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b81e4660 con 0x7f88b81e6200 2024-03-16T20:09:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f88bd1b1640 1 -- 172.21.15.32:0/1527542430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b80c5140 con 0x7f88b81e6200 2024-03-16T20:09:43.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.098+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88900052a0 con 0x7f88b81e6200 2024-03-16T20:09:43.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.130+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f88b0012420 con 0x7f88b81e6200 2024-03-16T20:09:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.134+0000 7f889f7fe640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 0x7f88a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.134+0000 7f88b6d9d640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 0x7f88a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.134+0000 7f88b6d9d640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 0x7f88a4069350 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88a80b2000 tx=0x7f88a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:43.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.418+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f88b0055fe0 con 0x7f88b81e6200 2024-03-16T20:09:43.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.418+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f88b0052280 con 0x7f88b81e6200 2024-03-16T20:09:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.698+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8890005900 con 0x7f88b81e6200 2024-03-16T20:09:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.718+0000 7f889f7fe640 1 -- 172.21.15.32:0/1527542430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8890005900 con 0x7f88b81e6200 2024-03-16T20:09:43.711 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:43.711 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 msgr2=0x7f88a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 0x7f88a4069350 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88a80b2000 tx=0x7f88a8009040 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 msgr2=0x7f88b81e4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c780 tx=0x7f88b000cc40 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 shutdown_connections 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f88a4066e90 0x7f88a4069350 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b81e6200 0x7f88b81e4e20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b815e250 0x7f88b815cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 --2- 172.21.15.32:0/1527542430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f88b80c8bb0 0x7f88b81505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 >> 172.21.15.32:0/1527542430 conn(0x7f88b80cc7a0 msgr2=0x7f88b806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 shutdown_connections 2024-03-16T20:09:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:43.722+0000 7f889d7fa640 1 -- 172.21.15.32:0/1527542430 wait complete. 2024-03-16T20:09:44.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:44.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:45.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:45.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 Processor -- start 2024-03-16T20:09:45.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 -- start start 2024-03-16T20:09:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fedac1521b0 0x7fedac157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac15b360 con 0x7fedac15bdb0 2024-03-16T20:09:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac15b920 con 0x7fedac06efc0 2024-03-16T20:09:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedb0843640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac0c44b0 con 0x7fedac1521b0 2024-03-16T20:09:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7feda9d9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedac15bdb0 0x7fedac0c82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59152/0 (socket says 172.21.15.32:59152) 2024-03-16T20:09:45.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7feda9d9b640 1 -- 172.21.15.32:0/1559709325 learned_addr learned my addr 172.21.15.32:0/1559709325 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:45.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.458+0000 7fedaa59c640 1 --2- 172.21.15.32:0/1559709325 >> v2:172.21.15.59:3300/0 conn(0x7fedac1521b0 0x7fedac157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.462+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001981788 0 0) 0x7fedac15b360 con 0x7fedac15bdb0 2024-03-16T20:09:45.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.462+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed94003530 con 0x7fedac15bdb0 2024-03-16T20:09:45.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.466+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2123095958 0 0) 0x7fed94003530 con 0x7fedac15bdb0 2024-03-16T20:09:45.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.466+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 >> v2:172.21.15.59:3300/0 conn(0x7fedac1521b0 msgr2=0x7fedac157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:45.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.466+0000 7feda959a640 1 --2- 172.21.15.32:0/1559709325 >> v2:172.21.15.59:3300/0 conn(0x7fedac1521b0 0x7fedac157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.466+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 >> v1:172.21.15.59:6789/0 conn(0x7fedac06efc0 legacy=0x7fedac162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:45.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.466+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac152590 con 0x7fedac15bdb0 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7feda0001680 con 0x7fedac15bdb0 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 >> v1:172.21.15.32:6789/0 conn(0x7fedac15bdb0 legacy=0x7fedac0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fed94003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fed94004330 0x7fed940047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed94004ff0 0x7fed9400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9400bf00 con 0x7fed94003980 2024-03-16T20:09:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9400c070 con 0x7fed94004ff0 2024-03-16T20:09:45.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.470+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9400c230 con 0x7fed94004330 2024-03-16T20:09:45.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.474+0000 7feda9d9b640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fed94004330 0x7fed940047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.474+0000 7fedaa59c640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fed94003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.474+0000 7fedaad9d640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed94004ff0 0x7fed9400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 -- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fed94004330 msgr2=0x7fed940047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:45.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fed94004330 0x7fed940047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 -- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed94004ff0 msgr2=0x7fed9400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed94004ff0 0x7fed9400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 -- 172.21.15.32:0/1559709325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94003530 con 0x7fed94003980 2024-03-16T20:09:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaad9d640 1 -- 172.21.15.32:0/1559709325 reap_dead start 2024-03-16T20:09:45.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.478+0000 7fedaa59c640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fed94003d60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009ae0 tx=0x7fed9c01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6273703388b456 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0a55f0 con 0x7fed94003980 2024-03-16T20:09:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c0a5750 con 0x7fed94003980 2024-03-16T20:09:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7feda959a640 1 -- 172.21.15.32:0/1559709325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0a5b40 con 0x7fed94003980 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 msgr2=0x7fed94003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fed94003d60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009ae0 tx=0x7fed9c01cb40 comp rx=0 tx=0).stop 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- 172.21.15.32:0/1559709325 shutdown_connections 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 --2- 172.21.15.32:0/1559709325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fed94003d60 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- 172.21.15.32:0/1559709325 >> 172.21.15.32:0/1559709325 conn(0x7fedac06b1c0 msgr2=0x7fedac066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- 172.21.15.32:0/1559709325 shutdown_connections 2024-03-16T20:09:45.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- 172.21.15.32:0/1559709325 wait complete. 2024-03-16T20:09:45.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 Processor -- start 2024-03-16T20:09:45.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- start start 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedac0c50a0 0x7fedac0c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedac0c58f0 0x7fedac14df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac1538f0 con 0x7fed94003980 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac1f1960 con 0x7fedac0c58f0 2024-03-16T20:09:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.486+0000 7fedb0843640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0c5f80 con 0x7fedac0c50a0 2024-03-16T20:09:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.490+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.490+0000 7fedaad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedac0c58f0 0x7fedac14df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.490+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58818/0 (socket says 172.21.15.32:58818) 2024-03-16T20:09:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.490+0000 7fedaa59c640 1 -- 172.21.15.32:0/3202262284 learned_addr learned my addr 172.21.15.32:0/3202262284 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:45.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.490+0000 7feda9d9b640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedac0c50a0 0x7fedac0c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.494+0000 7fedaa59c640 1 -- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedac0c50a0 msgr2=0x7fedac0c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.494+0000 7fedaa59c640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedac0c50a0 0x7fedac0c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.494+0000 7fedaa59c640 1 -- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedac0c58f0 msgr2=0x7fedac14df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.494+0000 7fedaa59c640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedac0c58f0 0x7fedac14df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.494+0000 7fedaa59c640 1 -- 172.21.15.32:0/3202262284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac0b17e0 con 0x7fed94003980 2024-03-16T20:09:45.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.498+0000 7fedaa59c640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fed9c01c970 tx=0x7fed94005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.502+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0ad850 con 0x7fed94003980 2024-03-16T20:09:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.502+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c0ad9f0 con 0x7fed94003980 2024-03-16T20:09:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.502+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0adda0 con 0x7fed94003980 2024-03-16T20:09:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.502+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac0b1ac0 con 0x7fed94003980 2024-03-16T20:09:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.502+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac0c6240 con 0x7fed94003980 2024-03-16T20:09:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.506+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac0c6460 con 0x7fed94003980 2024-03-16T20:09:45.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.530+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fed9c0b4040 con 0x7fed94003980 2024-03-16T20:09:45.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.530+0000 7fed9affd640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 0x7fed8c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:45.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.530+0000 7feda9d9b640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 0x7fed8c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:45.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.530+0000 7feda9d9b640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 0x7fed8c069350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fedac1f1c30 tx=0x7feda0001a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:45.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.534+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fed9c00f080 con 0x7fed94003980 2024-03-16T20:09:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:45.822+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fed9c0ebfb0 con 0x7fed94003980 2024-03-16T20:09:46.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.134+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedac068000 con 0x7fed94003980 2024-03-16T20:09:46.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fed9affd640 1 -- 172.21.15.32:0/3202262284 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fed9c0b2090 con 0x7fed94003980 2024-03-16T20:09:46.151 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:46.151 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:46.151 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 msgr2=0x7fed8c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 0x7fed8c069350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fedac1f1c30 tx=0x7feda0001a00 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 msgr2=0x7fedac0c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fed9c01c970 tx=0x7fed94005800 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 shutdown_connections 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fed8c066e90 0x7fed8c069350 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedac0c58f0 0x7fedac14df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fedac0c50a0 0x7fedac0c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 --2- 172.21.15.32:0/3202262284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed94003980 0x7fedac0c6760 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.158+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 >> 172.21.15.32:0/3202262284 conn(0x7fedac06b1c0 msgr2=0x7fedac0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:46.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.162+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 shutdown_connections 2024-03-16T20:09:46.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:46.162+0000 7fedb0843640 1 -- 172.21.15.32:0/3202262284 wait complete. 2024-03-16T20:09:46.774 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:46.774 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:47.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 Processor -- start 2024-03-16T20:09:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 -- start start 2024-03-16T20:09:47.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ae00cc590 0x7f8ae00cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ae015dbb0 0x7f8ae00ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae00cbfa0 con 0x7f8ae00caab0 2024-03-16T20:09:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00cceb0 con 0x7f8ae00cc590 2024-03-16T20:09:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ae585a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae015a140 con 0x7f8ae015dbb0 2024-03-16T20:09:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8adeffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ae00cc590 0x7f8ae00cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8ade7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ae015dbb0 0x7f8ae00ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8adeffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ae00cc590 0x7f8ae00cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58822/0 (socket says 172.21.15.32:58822) 2024-03-16T20:09:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.894+0000 7f8adeffd640 1 -- 172.21.15.32:0/3139504170 learned_addr learned my addr 172.21.15.32:0/3139504170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.898+0000 7f8adeffd640 1 -- 172.21.15.32:0/3139504170 >> v2:172.21.15.59:3300/0 conn(0x7f8ae015dbb0 msgr2=0x7f8ae00ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.898+0000 7f8adeffd640 1 --2- 172.21.15.32:0/3139504170 >> v2:172.21.15.59:3300/0 conn(0x7f8ae015dbb0 0x7f8ae00ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.898+0000 7f8adeffd640 1 -- 172.21.15.32:0/3139504170 >> v1:172.21.15.59:6789/0 conn(0x7f8ae00caab0 legacy=0x7f8ae006bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.898+0000 7f8adeffd640 1 -- 172.21.15.32:0/3139504170 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae015a3e0 con 0x7f8ae00cc590 2024-03-16T20:09:47.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.902+0000 7f8adeffd640 1 --2- 172.21.15.32:0/3139504170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae00cc590 0x7f8ae00cc970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00ccff0 tx=0x7f8ad001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afecfb5d1f29a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:47.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.906+0000 7f8addffb640 1 -- 172.21.15.32:0/3139504170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad00a89d0 con 0x7f8ae00cc590 2024-03-16T20:09:47.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8addffb640 1 -- 172.21.15.32:0/3139504170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad00a8b30 con 0x7f8ae00cc590 2024-03-16T20:09:47.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8addffb640 1 -- 172.21.15.32:0/3139504170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad00b0440 con 0x7f8ae00cc590 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- 172.21.15.32:0/3139504170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae00cc590 msgr2=0x7f8ae00cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- 172.21.15.32:0/3139504170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae00cc590 0x7f8ae00cc970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00ccff0 tx=0x7f8ad001ecd0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- 172.21.15.32:0/3139504170 shutdown_connections 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- 172.21.15.32:0/3139504170 >> v2:172.21.15.59:3300/0 conn(0x7f8ae015dbb0 0x7f8ae00ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- 172.21.15.32:0/3139504170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae00cc590 0x7f8ae00cc970 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- 172.21.15.32:0/3139504170 >> 172.21.15.32:0/3139504170 conn(0x7f8ae0066ea0 msgr2=0x7f8ae00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- 172.21.15.32:0/3139504170 shutdown_connections 2024-03-16T20:09:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- 172.21.15.32:0/3139504170 wait complete. 2024-03-16T20:09:47.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 Processor -- start 2024-03-16T20:09:47.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- start start 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ae00caab0 0x7f8ae00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ae00cc590 0x7f8ae014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00c63a0 con 0x7f8ae015dbb0 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae014cf70 con 0x7f8ae00caab0 2024-03-16T20:09:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.910+0000 7f8ae585a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01f3340 con 0x7f8ae00cc590 2024-03-16T20:09:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.914+0000 7f8adf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.914+0000 7f8adeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ae00caab0 0x7f8ae00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.914+0000 7f8adf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58824/0 (socket says 172.21.15.32:58824) 2024-03-16T20:09:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.914+0000 7f8adf7fe640 1 -- 172.21.15.32:0/2946065040 learned_addr learned my addr 172.21.15.32:0/2946065040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.914+0000 7f8ade7fc640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ae00cc590 0x7f8ae014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.918+0000 7f8adf7fe640 1 -- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ae00cc590 msgr2=0x7f8ae014b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.918+0000 7f8adf7fe640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ae00cc590 0x7f8ae014b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.918+0000 7f8adf7fe640 1 -- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ae00caab0 msgr2=0x7f8ae00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.918+0000 7f8adf7fe640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ae00caab0 0x7f8ae00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.918+0000 7f8adf7fe640 1 -- 172.21.15.32:0/2946065040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00094d0 con 0x7f8ae015dbb0 2024-03-16T20:09:47.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.922+0000 7f8adf7fe640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800c780 tx=0x7f8ad800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.926+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8003d90 con 0x7f8ae015dbb0 2024-03-16T20:09:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.926+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01f3ec0 con 0x7f8ae015dbb0 2024-03-16T20:09:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.926+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01f3590 con 0x7f8ae015dbb0 2024-03-16T20:09:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:47.926+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae006cef0 con 0x7f8ae015dbb0 2024-03-16T20:09:48.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8001790 con 0x7f8ae015dbb0 2024-03-16T20:09:48.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8005290 con 0x7f8ae015dbb0 2024-03-16T20:09:48.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8ad80054b0 con 0x7f8ae015dbb0 2024-03-16T20:09:48.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8ac7fff640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 0x7f8acc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:48.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8adeffd640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 0x7f8acc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:48.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.238+0000 7f8adeffd640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 0x7f8acc069300 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00b2000 tx=0x7f8ad00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:48.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.262+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8ad8055270 con 0x7f8ae015dbb0 2024-03-16T20:09:48.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.262+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8ad8051510 con 0x7f8ae015dbb0 2024-03-16T20:09:48.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.458+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae01f4050 con 0x7f8ae015dbb0 2024-03-16T20:09:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.482+0000 7f8ac7fff640 1 -- 172.21.15.32:0/2946065040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8ad8021090 con 0x7f8ae015dbb0 2024-03-16T20:09:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:48.474 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 msgr2=0x7f8acc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 0x7f8acc069300 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00b2000 tx=0x7f8ad00aa070 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 msgr2=0x7f8ae01f1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800c780 tx=0x7f8ad800cc40 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 shutdown_connections 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8acc066e40 0x7f8acc069300 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae015dbb0 0x7f8ae01f1240 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ae00cc590 0x7f8ae014b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 --2- 172.21.15.32:0/2946065040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ae00caab0 0x7f8ae00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 >> 172.21.15.32:0/2946065040 conn(0x7f8ae0066ea0 msgr2=0x7f8ae01583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 shutdown_connections 2024-03-16T20:09:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:48.486+0000 7f8ae585a640 1 -- 172.21.15.32:0/2946065040 wait complete. 2024-03-16T20:09:49.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:49.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:50.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 Processor -- start 2024-03-16T20:09:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 -- start start 2024-03-16T20:09:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff8400714f0 0x7ff8400caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400c9d10 con 0x7ff840071110 2024-03-16T20:09:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400ca8e0 con 0x7ff8401544f0 2024-03-16T20:09:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff844cdd640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400cab80 con 0x7ff8400714f0 2024-03-16T20:09:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff83ed9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff8400714f0 0x7ff8400caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff83dd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff8401544f0 0x7ff84015a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:44464/0 (socket says 172.21.15.32:44464) 2024-03-16T20:09:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.214+0000 7ff83dd9b640 1 -- 172.21.15.32:0/757965158 learned_addr learned my addr 172.21.15.32:0/757965158 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.218+0000 7ff83ed9d640 1 -- 172.21.15.32:0/757965158 >> v1:172.21.15.59:6789/0 conn(0x7ff8401544f0 legacy=0x7ff84015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.218+0000 7ff83ed9d640 1 -- 172.21.15.32:0/757965158 >> v1:172.21.15.32:6789/0 conn(0x7ff840071110 legacy=0x7ff840153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.218+0000 7ff83ed9d640 1 -- 172.21.15.32:0/757965158 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400c9f60 con 0x7ff8400714f0 2024-03-16T20:09:50.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.226+0000 7ff83ed9d640 1 --2- 172.21.15.32:0/757965158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8400caf70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff838009d90 tx=0x7ff83801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5638e20d881f9f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:50.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff83d59a640 1 -- 172.21.15.32:0/757965158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380a5a20 con 0x7ff8400714f0 2024-03-16T20:09:50.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff83d59a640 1 -- 172.21.15.32:0/757965158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8380a5b80 con 0x7ff8400714f0 2024-03-16T20:09:50.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff83d59a640 1 -- 172.21.15.32:0/757965158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380ad440 con 0x7ff8400714f0 2024-03-16T20:09:50.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- 172.21.15.32:0/757965158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 msgr2=0x7ff8400caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 --2- 172.21.15.32:0/757965158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8400caf70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff838009d90 tx=0x7ff83801c940 comp rx=0 tx=0).stop 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- 172.21.15.32:0/757965158 shutdown_connections 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 --2- 172.21.15.32:0/757965158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8400caf70 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- 172.21.15.32:0/757965158 >> 172.21.15.32:0/757965158 conn(0x7ff84006afc0 msgr2=0x7ff84006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- 172.21.15.32:0/757965158 shutdown_connections 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- 172.21.15.32:0/757965158 wait complete. 2024-03-16T20:09:50.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 Processor -- start 2024-03-16T20:09:50.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- start start 2024-03-16T20:09:50.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 0x7ff8400c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:50.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8401f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8401544f0 0x7ff840156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8401513d0 con 0x7ff8400714f0 2024-03-16T20:09:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff840150f40 con 0x7ff8401544f0 2024-03-16T20:09:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.230+0000 7ff844cdd640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400c5760 con 0x7ff840071110 2024-03-16T20:09:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.234+0000 7ff83dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8401f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.234+0000 7ff83ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8401544f0 0x7ff840156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.234+0000 7ff83dd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8401f6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46520/0 (socket says 172.21.15.32:46520) 2024-03-16T20:09:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.234+0000 7ff83dd9b640 1 -- 172.21.15.32:0/1385497739 learned_addr learned my addr 172.21.15.32:0/1385497739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.234+0000 7ff83e59c640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 0x7ff8400c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.238+0000 7ff83e59c640 1 -- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8401544f0 msgr2=0x7ff840156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.238+0000 7ff83e59c640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8401544f0 0x7ff840156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.238+0000 7ff83e59c640 1 -- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 msgr2=0x7ff8401f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.238+0000 7ff83e59c640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8401f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.238+0000 7ff83e59c640 1 -- 172.21.15.32:0/1385497739 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838009a70 con 0x7ff840071110 2024-03-16T20:09:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.270+0000 7ff83e59c640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 0x7ff8400c3920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff83000c3f0 tx=0x7ff83000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:50.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8300052d0 con 0x7ff840071110 2024-03-16T20:09:50.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830005430 con 0x7ff840071110 2024-03-16T20:09:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8400c5a40 con 0x7ff840071110 2024-03-16T20:09:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830007260 con 0x7ff840071110 2024-03-16T20:09:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8401508f0 con 0x7ff840071110 2024-03-16T20:09:50.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.278+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840068000 con 0x7ff840071110 2024-03-16T20:09:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.570+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff8300073c0 con 0x7ff840071110 2024-03-16T20:09:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.570+0000 7ff82effd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 0x7ff820069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.570+0000 7ff83dd9b640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 0x7ff820069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.570+0000 7ff83dd9b640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 0x7ff820069330 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff840152c50 tx=0x7ff834009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.574+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff830007670 con 0x7ff840071110 2024-03-16T20:09:50.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.586+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff830051430 con 0x7ff840071110 2024-03-16T20:09:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.806+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84006b680 con 0x7ff840071110 2024-03-16T20:09:50.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.842+0000 7ff82effd640 1 -- 172.21.15.32:0/1385497739 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7ff830020020 con 0x7ff840071110 2024-03-16T20:09:50.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:50.834 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:50.834 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:50.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 msgr2=0x7ff820069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 0x7ff820069330 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff840152c50 tx=0x7ff834009040 comp rx=0 tx=0).stop 2024-03-16T20:09:50.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 msgr2=0x7ff8400c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 0x7ff8400c3920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff83000c3f0 tx=0x7ff83000c8b0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 shutdown_connections 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff820066e70 0x7ff820069330 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8401544f0 0x7ff840156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8400714f0 0x7ff8401f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 --2- 172.21.15.32:0/1385497739 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff840071110 0x7ff8400c3920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 >> 172.21.15.32:0/1385497739 conn(0x7ff84006afc0 msgr2=0x7ff840151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 shutdown_connections 2024-03-16T20:09:50.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:50.846+0000 7ff844cdd640 1 -- 172.21.15.32:0/1385497739 wait complete. 2024-03-16T20:09:51.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:51.458 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:52.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 Processor -- start 2024-03-16T20:09:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 -- start start 2024-03-16T20:09:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23ac154350 0x7f23ac06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23ac0c2d70 0x7f23ac15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac072a10 0x7f23ac072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac06bae0 con 0x7f23ac0c2d70 2024-03-16T20:09:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac15a540 con 0x7f23ac154350 2024-03-16T20:09:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.574+0000 7f23b11b7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0c7fd0 con 0x7f23ac072a10 2024-03-16T20:09:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.578+0000 7f23aa59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23ac0c2d70 0x7f23ac15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.578+0000 7f23ab59e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac072a10 0x7f23ac072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.578+0000 7f23aad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23ac154350 0x7f23ac06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.578+0000 7f23aa59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23ac0c2d70 0x7f23ac15a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46526/0 (socket says 172.21.15.32:46526) 2024-03-16T20:09:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.578+0000 7f23aa59c640 1 -- 172.21.15.32:0/3333491297 learned_addr learned my addr 172.21.15.32:0/3333491297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 -- 172.21.15.32:0/3333491297 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac072a10 msgr2=0x7f23ac072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 --2- 172.21.15.32:0/3333491297 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac072a10 0x7f23ac072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 -- 172.21.15.32:0/3333491297 >> v2:172.21.15.59:3300/0 conn(0x7f23ac154350 msgr2=0x7f23ac06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 --2- 172.21.15.32:0/3333491297 >> v2:172.21.15.59:3300/0 conn(0x7f23ac154350 0x7f23ac06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 -- 172.21.15.32:0/3333491297 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac06bd10 con 0x7f23ac0c2d70 2024-03-16T20:09:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.582+0000 7f23aa59c640 1 --2- 172.21.15.32:0/3333491297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac0c2d70 0x7f23ac15a9d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0c9a80 tx=0x7f23a001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d00eb225a980578 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23a9d9b640 1 -- 172.21.15.32:0/3333491297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00ab600 con 0x7f23ac0c2d70 2024-03-16T20:09:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23a9d9b640 1 -- 172.21.15.32:0/3333491297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a00a7050 con 0x7f23ac0c2d70 2024-03-16T20:09:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23a9d9b640 1 -- 172.21.15.32:0/3333491297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00ab980 con 0x7f23ac0c2d70 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- 172.21.15.32:0/3333491297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac0c2d70 msgr2=0x7f23ac15a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 --2- 172.21.15.32:0/3333491297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac0c2d70 0x7f23ac15a9d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0c9a80 tx=0x7f23a001c470 comp rx=0 tx=0).stop 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- 172.21.15.32:0/3333491297 shutdown_connections 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 --2- 172.21.15.32:0/3333491297 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac072a10 0x7f23ac072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 --2- 172.21.15.32:0/3333491297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac0c2d70 0x7f23ac15a9d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 --2- 172.21.15.32:0/3333491297 >> v2:172.21.15.59:3300/0 conn(0x7f23ac154350 0x7f23ac06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- 172.21.15.32:0/3333491297 >> 172.21.15.32:0/3333491297 conn(0x7f23ac06afc0 msgr2=0x7f23ac06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- 172.21.15.32:0/3333491297 shutdown_connections 2024-03-16T20:09:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- 172.21.15.32:0/3333491297 wait complete. 2024-03-16T20:09:52.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 Processor -- start 2024-03-16T20:09:52.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.590+0000 7f23b11b7640 1 -- start start 2024-03-16T20:09:52.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23ac0c2d70 0x7f23ac152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac154350 0x7f23ac1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac156c70 con 0x7f23ac072a10 2024-03-16T20:09:52.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac151a10 con 0x7f23ac0c2d70 2024-03-16T20:09:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23b11b7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac152fd0 con 0x7f23ac154350 2024-03-16T20:09:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23aad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23aad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46530/0 (socket says 172.21.15.32:46530) 2024-03-16T20:09:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23aad9d640 1 -- 172.21.15.32:0/364952758 learned_addr learned my addr 172.21.15.32:0/364952758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23ab59e640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac154350 0x7f23ac1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.594+0000 7f23aa59c640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23ac0c2d70 0x7f23ac152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.602+0000 7f23aad9d640 1 -- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac154350 msgr2=0x7f23ac1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.602+0000 7f23aad9d640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac154350 0x7f23ac1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.602+0000 7f23aad9d640 1 -- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23ac0c2d70 msgr2=0x7f23ac152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.602+0000 7f23aad9d640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23ac0c2d70 0x7f23ac152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:52.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.602+0000 7f23aad9d640 1 -- 172.21.15.32:0/364952758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00094d0 con 0x7f23ac072a10 2024-03-16T20:09:52.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.606+0000 7f23aad9d640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f239c00e870 tx=0x7f239c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:52.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c009510 con 0x7f23ac072a10 2024-03-16T20:09:52.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac1f33a0 con 0x7f23ac072a10 2024-03-16T20:09:52.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac151c60 con 0x7f23ac072a10 2024-03-16T20:09:52.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac0674e0 con 0x7f23ac072a10 2024-03-16T20:09:52.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c00eec0 con 0x7f23ac072a10 2024-03-16T20:09:52.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.610+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c005bc0 con 0x7f23ac072a10 2024-03-16T20:09:52.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.914+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f239c00c940 con 0x7f23ac072a10 2024-03-16T20:09:52.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.914+0000 7f23937fe640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 0x7f2398069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.914+0000 7f23aa59c640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 0x7f2398069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.914+0000 7f23aa59c640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 0x7f2398069300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23a00ad000 tx=0x7f23a00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:52.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.938+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f239c055ee0 con 0x7f23ac072a10 2024-03-16T20:09:52.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:52.938+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f239c052180 con 0x7f23ac072a10 2024-03-16T20:09:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.158+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23ac068000 con 0x7f23ac072a10 2024-03-16T20:09:53.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.182+0000 7f23937fe640 1 -- 172.21.15.32:0/364952758 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f239c021090 con 0x7f23ac072a10 2024-03-16T20:09:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:53.174 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:53.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 msgr2=0x7f2398069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:53.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 0x7f2398069300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23a00ad000 tx=0x7f23a00a6000 comp rx=0 tx=0).stop 2024-03-16T20:09:53.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 msgr2=0x7f23ac15c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f239c00e870 tx=0x7f239c00ec70 comp rx=0 tx=0).stop 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 shutdown_connections 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2398066e40 0x7f2398069300 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f23ac154350 0x7f23ac1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23ac0c2d70 0x7f23ac152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 --2- 172.21.15.32:0/364952758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23ac072a10 0x7f23ac15c6e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 >> 172.21.15.32:0/364952758 conn(0x7f23ac06afc0 msgr2=0x7f23ac159300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 shutdown_connections 2024-03-16T20:09:53.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:53.186+0000 7f23b11b7640 1 -- 172.21.15.32:0/364952758 wait complete. 2024-03-16T20:09:53.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:53.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:54.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:54.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 Processor -- start 2024-03-16T20:09:54.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 -- start start 2024-03-16T20:09:54.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd9f80c81e0 0x7fd9f80c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:54.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f806c460 con 0x7fd9f80c3570 2024-03-16T20:09:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f815a240 con 0x7fd9f81540c0 2024-03-16T20:09:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fe335640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f806cf20 con 0x7fd9f80c81e0 2024-03-16T20:09:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fc8d2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd9f80c3570 0x7fd9f81647e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52312/0 (socket says 172.21.15.32:52312) 2024-03-16T20:09:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.946+0000 7fd9fc8d2640 1 -- 172.21.15.32:0/3918269722 learned_addr learned my addr 172.21.15.32:0/3918269722 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:09:54.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.950+0000 7fd9f77fe640 1 --2- 172.21.15.32:0/3918269722 >> v2:172.21.15.59:3300/0 conn(0x7fd9f80c81e0 0x7fd9f80c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.954+0000 7fd9f6ffd640 1 -- 172.21.15.32:0/3918269722 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197447840 0 0) 0x7fd9f806c460 con 0x7fd9f80c3570 2024-03-16T20:09:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.954+0000 7fd9f6ffd640 1 -- 172.21.15.32:0/3918269722 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e0003530 con 0x7fd9f80c3570 2024-03-16T20:09:54.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.958+0000 7fd9f77fe640 1 -- 172.21.15.32:0/3918269722 >> v1:172.21.15.59:6789/0 conn(0x7fd9f81540c0 legacy=0x7fd9f815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:54.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.958+0000 7fd9f77fe640 1 -- 172.21.15.32:0/3918269722 >> v1:172.21.15.32:6789/0 conn(0x7fd9f80c3570 legacy=0x7fd9f81647e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:54.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.958+0000 7fd9f77fe640 1 -- 172.21.15.32:0/3918269722 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f806d200 con 0x7fd9f80c81e0 2024-03-16T20:09:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.966+0000 7fd9f77fe640 1 --2- 172.21.15.32:0/3918269722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f80c8630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0097c0 tx=0x7fd9ec01c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b288483b585e1805 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:54.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.970+0000 7fd9f6ffd640 1 -- 172.21.15.32:0/3918269722 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec0a7070 con 0x7fd9f80c81e0 2024-03-16T20:09:54.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9f6ffd640 1 -- 172.21.15.32:0/3918269722 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec0ab430 con 0x7fd9f80c81e0 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 -- 172.21.15.32:0/3918269722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 msgr2=0x7fd9f80c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3918269722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f80c8630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0097c0 tx=0x7fd9ec01c430 comp rx=0 tx=0).stop 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 -- 172.21.15.32:0/3918269722 shutdown_connections 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3918269722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f80c8630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 -- 172.21.15.32:0/3918269722 >> 172.21.15.32:0/3918269722 conn(0x7fd9f8070f90 msgr2=0x7fd9f80713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 -- 172.21.15.32:0/3918269722 shutdown_connections 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.974+0000 7fd9fe335640 1 -- 172.21.15.32:0/3918269722 wait complete. 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 Processor -- start 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 -- start start 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9f80c3570 0x7fd9f8143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:54.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f8140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f81444e0 con 0x7fd9f81540c0 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f81584d0 con 0x7fd9f80c81e0 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fe335640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80cca30 con 0x7fd9f80c3570 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fc8d2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fc8d2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46538/0 (socket says 172.21.15.32:46538) 2024-03-16T20:09:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9fc8d2640 1 -- 172.21.15.32:0/3589710922 learned_addr learned my addr 172.21.15.32:0/3589710922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9f7fff640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9f80c3570 0x7fd9f8143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:54.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.978+0000 7fd9f77fe640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f8140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.982+0000 7fd9fc8d2640 1 -- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9f80c3570 msgr2=0x7fd9f8143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.982+0000 7fd9fc8d2640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9f80c3570 0x7fd9f8143d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.982+0000 7fd9fc8d2640 1 -- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 msgr2=0x7fd9f8140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.982+0000 7fd9fc8d2640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f8140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.982+0000 7fd9fc8d2640 1 -- 172.21.15.32:0/3589710922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec009420 con 0x7fd9f81540c0 2024-03-16T20:09:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.986+0000 7fd9fc8d2640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000cb40 tx=0x7fd9f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:54.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f000aca0 con 0x7fd9f81540c0 2024-03-16T20:09:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f815b1c0 con 0x7fd9f81540c0 2024-03-16T20:09:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f80cbff0 con 0x7fd9f81540c0 2024-03-16T20:09:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f0001c30 con 0x7fd9f81540c0 2024-03-16T20:09:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f001c640 con 0x7fd9f81540c0 2024-03-16T20:09:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:54.990+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f814c8a0 con 0x7fd9f81540c0 2024-03-16T20:09:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.018+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd9f001c7a0 con 0x7fd9f81540c0 2024-03-16T20:09:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.018+0000 7fd9f4ff9640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 0x7fd9e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.018+0000 7fd9f7fff640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 0x7fd9e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.018+0000 7fd9f7fff640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 0x7fd9e4069300 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80098e0 tx=0x7fd9e8009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:55.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.278+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd9f000ae40 con 0x7fd9f81540c0 2024-03-16T20:09:55.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.278+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd9f0051ee0 con 0x7fd9f81540c0 2024-03-16T20:09:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.542+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9f814c8a0 con 0x7fd9f81540c0 2024-03-16T20:09:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.566+0000 7fd9f4ff9640 1 -- 172.21.15.32:0/3589710922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fd9f814c8a0 con 0x7fd9f81540c0 2024-03-16T20:09:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:55.560 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 msgr2=0x7fd9e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 0x7fd9e4069300 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80098e0 tx=0x7fd9e8009060 comp rx=0 tx=0).stop 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 msgr2=0x7fd9f814cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000cb40 tx=0x7fd9f000a060 comp rx=0 tx=0).stop 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 shutdown_connections 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd9e4066e40 0x7fd9e4069300 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9f81540c0 0x7fd9f814cb80 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9f80c81e0 0x7fd9f8140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 --2- 172.21.15.32:0/3589710922 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9f80c3570 0x7fd9f8143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 >> 172.21.15.32:0/3589710922 conn(0x7fd9f8070f90 msgr2=0x7fd9f814b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 shutdown_connections 2024-03-16T20:09:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:55.570+0000 7fd9fe335640 1 -- 172.21.15.32:0/3589710922 wait complete. 2024-03-16T20:09:56.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:56.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:57.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:57.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 Processor -- start 2024-03-16T20:09:57.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 -- start start 2024-03-16T20:09:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd940154350 0x7fd94006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9400c2d70 0x7fd94015a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd940072a10 0x7fd940072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94006bae0 con 0x7fd9400c2d70 2024-03-16T20:09:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94015a540 con 0x7fd940154350 2024-03-16T20:09:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.306+0000 7fd946b8c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9400c7fd0 con 0x7fd940072a10 2024-03-16T20:09:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.310+0000 7fd93ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9400c2d70 0x7fd94015a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.310+0000 7fd944928640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd940154350 0x7fd94006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.310+0000 7fd93ffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9400c2d70 0x7fd94015a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46548/0 (socket says 172.21.15.32:46548) 2024-03-16T20:09:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.310+0000 7fd93ffff640 1 -- 172.21.15.32:0/2869766618 learned_addr learned my addr 172.21.15.32:0/2869766618 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.310+0000 7fd945129640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd940072a10 0x7fd940072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.314+0000 7fd93ffff640 1 -- 172.21.15.32:0/2869766618 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd940072a10 msgr2=0x7fd940072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.314+0000 7fd93ffff640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd940072a10 0x7fd940072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.314+0000 7fd93ffff640 1 -- 172.21.15.32:0/2869766618 >> v2:172.21.15.59:3300/0 conn(0x7fd940154350 msgr2=0x7fd94006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.314+0000 7fd93ffff640 1 --2- 172.21.15.32:0/2869766618 >> v2:172.21.15.59:3300/0 conn(0x7fd940154350 0x7fd94006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.314+0000 7fd93ffff640 1 -- 172.21.15.32:0/2869766618 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94006bd10 con 0x7fd9400c2d70 2024-03-16T20:09:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.318+0000 7fd93ffff640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9400c2d70 0x7fd94015a9d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd9400c9a80 tx=0x7fd92c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c76d8eecb94629 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:57.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd93f7fe640 1 -- 172.21.15.32:0/2869766618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0ab600 con 0x7fd9400c2d70 2024-03-16T20:09:57.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd93f7fe640 1 -- 172.21.15.32:0/2869766618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c0a7050 con 0x7fd9400c2d70 2024-03-16T20:09:57.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd93f7fe640 1 -- 172.21.15.32:0/2869766618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0ab980 con 0x7fd9400c2d70 2024-03-16T20:09:57.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 -- 172.21.15.32:0/2869766618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9400c2d70 msgr2=0x7fd94015a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9400c2d70 0x7fd94015a9d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd9400c9a80 tx=0x7fd92c01c470 comp rx=0 tx=0).stop 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 -- 172.21.15.32:0/2869766618 shutdown_connections 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd940072a10 0x7fd940072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 --2- 172.21.15.32:0/2869766618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9400c2d70 0x7fd94015a9d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 --2- 172.21.15.32:0/2869766618 >> v2:172.21.15.59:3300/0 conn(0x7fd940154350 0x7fd94006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 -- 172.21.15.32:0/2869766618 >> 172.21.15.32:0/2869766618 conn(0x7fd94006afc0 msgr2=0x7fd94006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 -- 172.21.15.32:0/2869766618 shutdown_connections 2024-03-16T20:09:57.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.322+0000 7fd946b8c640 1 -- 172.21.15.32:0/2869766618 wait complete. 2024-03-16T20:09:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 Processor -- start 2024-03-16T20:09:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 -- start start 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9400c2d70 0x7fd940152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd940154350 0x7fd9401526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940156c70 con 0x7fd940072a10 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940151a10 con 0x7fd940154350 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd946b8c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd940152fd0 con 0x7fd9400c2d70 2024-03-16T20:09:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd944928640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd944928640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46558/0 (socket says 172.21.15.32:46558) 2024-03-16T20:09:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd944928640 1 -- 172.21.15.32:0/3647517409 learned_addr learned my addr 172.21.15.32:0/3647517409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.326+0000 7fd93ffff640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9400c2d70 0x7fd940152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.330+0000 7fd945129640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd940154350 0x7fd9401526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.334+0000 7fd944928640 1 -- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9400c2d70 msgr2=0x7fd940152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.334+0000 7fd944928640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9400c2d70 0x7fd940152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.334+0000 7fd944928640 1 -- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd940154350 msgr2=0x7fd9401526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.334+0000 7fd944928640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd940154350 0x7fd9401526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.334+0000 7fd944928640 1 -- 172.21.15.32:0/3647517409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c0094d0 con 0x7fd940072a10 2024-03-16T20:09:57.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.338+0000 7fd944928640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd93400d5f0 tx=0x7fd93400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:57.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9340125f0 con 0x7fd940072a10 2024-03-16T20:09:57.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd934012750 con 0x7fd940072a10 2024-03-16T20:09:57.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9401f33a0 con 0x7fd940072a10 2024-03-16T20:09:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934012ae0 con 0x7fd940072a10 2024-03-16T20:09:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd940151c60 con 0x7fd940072a10 2024-03-16T20:09:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.342+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9400674e0 con 0x7fd940072a10 2024-03-16T20:09:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.366+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd934012c40 con 0x7fd940072a10 2024-03-16T20:09:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.366+0000 7fd93d7fa640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 0x7fd924069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:57.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.366+0000 7fd93ffff640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 0x7fd924069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:57.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.370+0000 7fd93ffff640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 0x7fd924069300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0ad000 tx=0x7fd92c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.370+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd934056520 con 0x7fd940072a10 2024-03-16T20:09:57.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.658+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd9340527c0 con 0x7fd940072a10 2024-03-16T20:09:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.858+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd940068000 con 0x7fd940072a10 2024-03-16T20:09:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.930+0000 7fd93d7fa640 1 -- 172.21.15.32:0/3647517409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fd934055e80 con 0x7fd940072a10 2024-03-16T20:09:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:09:57.923 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:09:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:09:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 msgr2=0x7fd924069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 0x7fd924069300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0ad000 tx=0x7fd92c0a6000 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 msgr2=0x7fd94015c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd93400d5f0 tx=0x7fd93400dab0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 shutdown_connections 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066e40 0x7fd924069300 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd940154350 0x7fd9401526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd9400c2d70 0x7fd940152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 --2- 172.21.15.32:0/3647517409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd940072a10 0x7fd94015c6e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 >> 172.21.15.32:0/3647517409 conn(0x7fd94006afc0 msgr2=0x7fd940159300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 shutdown_connections 2024-03-16T20:09:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:57.934+0000 7fd946b8c640 1 -- 172.21.15.32:0/3647517409 wait complete. 2024-03-16T20:09:58.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:09:58.547 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:09:59.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:09:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.662+0000 7f8c079f5640 1 Processor -- start 2024-03-16T20:09:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.662+0000 7f8c079f5640 1 -- start start 2024-03-16T20:09:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8c00151f60 0x7f8c0006d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c0006da90 0x7f8c00158590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c000c6820 0x7f8c000c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c00152340 con 0x7f8c000c6820 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c00158100 con 0x7f8c00151f60 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c079f5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0006ed70 con 0x7f8c0006da90 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c05f92640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c000c6820 0x7f8c000c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c04f90640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c0006da90 0x7f8c00158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c05f92640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c000c6820 0x7f8c000c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37326/0 (socket says 172.21.15.32:37326) 2024-03-16T20:09:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c05f92640 1 -- 172.21.15.32:0/140898422 learned_addr learned my addr 172.21.15.32:0/140898422 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.666+0000 7f8c05791640 1 --2- 172.21.15.32:0/140898422 >> v2:172.21.15.59:3300/0 conn(0x7f8c00151f60 0x7f8c0006d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.670+0000 7f8c05f92640 1 -- 172.21.15.32:0/140898422 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c0006da90 msgr2=0x7f8c00158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.670+0000 7f8c05f92640 1 --2- 172.21.15.32:0/140898422 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c0006da90 0x7f8c00158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.670+0000 7f8c05f92640 1 -- 172.21.15.32:0/140898422 >> v2:172.21.15.59:3300/0 conn(0x7f8c00151f60 msgr2=0x7f8c0006d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:09:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.670+0000 7f8c05f92640 1 --2- 172.21.15.32:0/140898422 >> v2:172.21.15.59:3300/0 conn(0x7f8c00151f60 0x7f8c0006d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.670+0000 7f8c05f92640 1 -- 172.21.15.32:0/140898422 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0006e4d0 con 0x7f8c000c6820 2024-03-16T20:09:59.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.674+0000 7f8c05f92640 1 --2- 172.21.15.32:0/140898422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c000c6820 0x7f8c000c6c70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8c000705b0 tx=0x7f8bf801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab133985ca00adc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:59.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8bf7fff640 1 -- 172.21.15.32:0/140898422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf80af400 con 0x7f8c000c6820 2024-03-16T20:09:59.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8bf7fff640 1 -- 172.21.15.32:0/140898422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf80ab050 con 0x7f8c000c6820 2024-03-16T20:09:59.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8bf7fff640 1 -- 172.21.15.32:0/140898422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf80af780 con 0x7f8c000c6820 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 -- 172.21.15.32:0/140898422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c000c6820 msgr2=0x7f8c000c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 --2- 172.21.15.32:0/140898422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c000c6820 0x7f8c000c6c70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8c000705b0 tx=0x7f8bf801f1e0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 -- 172.21.15.32:0/140898422 shutdown_connections 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 --2- 172.21.15.32:0/140898422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c000c6820 0x7f8c000c6c70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 --2- 172.21.15.32:0/140898422 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c0006da90 0x7f8c00158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 --2- 172.21.15.32:0/140898422 >> v2:172.21.15.59:3300/0 conn(0x7f8c00151f60 0x7f8c0006d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 -- 172.21.15.32:0/140898422 >> 172.21.15.32:0/140898422 conn(0x7f8c000c8d20 msgr2=0x7f8c000c9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.678+0000 7f8c079f5640 1 -- 172.21.15.32:0/140898422 shutdown_connections 2024-03-16T20:09:59.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 -- 172.21.15.32:0/140898422 wait complete. 2024-03-16T20:09:59.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 Processor -- start 2024-03-16T20:09:59.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 -- start start 2024-03-16T20:09:59.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c0006da90 0x7f8c0015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c000c6820 0x7f8c000c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:09:59.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000c5260 con 0x7f8c00151f60 2024-03-16T20:09:59.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c001ed4e0 con 0x7f8c0006da90 2024-03-16T20:09:59.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.682+0000 7f8c079f5640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c001ed7c0 con 0x7f8c000c6820 2024-03-16T20:09:59.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.686+0000 7f8c05f92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.686+0000 7f8c04f90640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c000c6820 0x7f8c000c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.686+0000 7f8c05791640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c0006da90 0x7f8c0015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:09:59.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.686+0000 7f8c05f92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37334/0 (socket says 172.21.15.32:37334) 2024-03-16T20:09:59.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.686+0000 7f8c05f92640 1 -- 172.21.15.32:0/1614121430 learned_addr learned my addr 172.21.15.32:0/1614121430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:09:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.690+0000 7f8c05f92640 1 -- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c000c6820 msgr2=0x7f8c000c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.690+0000 7f8c05f92640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c000c6820 0x7f8c000c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.690+0000 7f8c05f92640 1 -- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c0006da90 msgr2=0x7f8c0015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:09:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.690+0000 7f8c05f92640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c0006da90 0x7f8c0015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:09:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.690+0000 7f8c05f92640 1 -- 172.21.15.32:0/1614121430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf8009b30 con 0x7f8c00151f60 2024-03-16T20:09:59.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.694+0000 7f8c05f92640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80b2040 tx=0x7f8bf801bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:09:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf80afc00 con 0x7f8c00151f60 2024-03-16T20:09:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf80afd80 con 0x7f8c00151f60 2024-03-16T20:09:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c001edaa0 con 0x7f8c00151f60 2024-03-16T20:09:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf80ba660 con 0x7f8c00151f60 2024-03-16T20:09:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c001ee060 con 0x7f8c00151f60 2024-03-16T20:09:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:09:59.698+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c001ee250 con 0x7f8c00151f60 2024-03-16T20:10:00.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.014+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8bf80b6040 con 0x7f8c00151f60 2024-03-16T20:10:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.014+0000 7f8bf5ffb640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 0x7f8be4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.014+0000 7f8c05791640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 0x7f8be4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.014+0000 7f8c05791640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 0x7f8be4069380 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009b60 tx=0x7f8bfc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:00.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.038+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8bf80f4040 con 0x7f8c00151f60 2024-03-16T20:10:00.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.038+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8bf80f02e0 con 0x7f8c00151f60 2024-03-16T20:10:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.382+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c00068000 con 0x7f8c00151f60 2024-03-16T20:10:00.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.398+0000 7f8bf5ffb640 1 -- 172.21.15.32:0/1614121430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8bf8009510 con 0x7f8c00151f60 2024-03-16T20:10:00.391 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:00.392 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 msgr2=0x7f8be4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 0x7f8be4069380 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009b60 tx=0x7f8bfc009190 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 msgr2=0x7f8c0014c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80b2040 tx=0x7f8bf801bf80 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 shutdown_connections 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8be4066ec0 0x7f8be4069380 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c00151f60 0x7f8c0014c160 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8c000c6820 0x7f8c000c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 --2- 172.21.15.32:0/1614121430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c0006da90 0x7f8c0015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 >> 172.21.15.32:0/1614121430 conn(0x7f8c000c8d20 msgr2=0x7f8c000c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 shutdown_connections 2024-03-16T20:10:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:00.402+0000 7f8c079f5640 1 -- 172.21.15.32:0/1614121430 wait complete. 2024-03-16T20:10:01.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:01.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:02.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:02.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.142+0000 7f8a31764640 1 Processor -- start 2024-03-16T20:10:02.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.142+0000 7f8a31764640 1 -- start start 2024-03-16T20:10:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a31764640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a2c0c16a0 0x7f8a2c151330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a31764640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c06e6a0 con 0x7f8a2c073050 2024-03-16T20:10:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a31764640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c06c250 con 0x7f8a2c072c70 2024-03-16T20:10:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a31764640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c156f20 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a2b7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a2c0c16a0 0x7f8a2c151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a2affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8a2c072c70 0x7f8a2c0c0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:43718/0 (socket says 172.21.15.32:43718) 2024-03-16T20:10:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.146+0000 7f8a2affd640 1 -- 172.21.15.32:0/2872378005 learned_addr learned my addr 172.21.15.32:0/2872378005 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.150+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/2872378005 >> v1:172.21.15.59:6789/0 conn(0x7f8a2c072c70 legacy=0x7f8a2c0c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.150+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/2872378005 >> v1:172.21.15.32:6789/0 conn(0x7f8a2c073050 legacy=0x7f8a2c150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.150+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/2872378005 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c06c510 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.154+0000 7f8a2b7fe640 1 --2- 172.21.15.32:0/2872378005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c151330 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009d90 tx=0x7f8a2401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cfc63b60828762 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.158+0000 7f8a29ffb640 1 -- 172.21.15.32:0/2872378005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240a5a00 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.158+0000 7f8a29ffb640 1 -- 172.21.15.32:0/2872378005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a240a5b60 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.158+0000 7f8a29ffb640 1 -- 172.21.15.32:0/2872378005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240ad440 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- 172.21.15.32:0/2872378005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 msgr2=0x7f8a2c151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 --2- 172.21.15.32:0/2872378005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c151330 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009d90 tx=0x7f8a2401c940 comp rx=0 tx=0).stop 2024-03-16T20:10:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- 172.21.15.32:0/2872378005 shutdown_connections 2024-03-16T20:10:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 --2- 172.21.15.32:0/2872378005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c151330 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- 172.21.15.32:0/2872378005 >> 172.21.15.32:0/2872378005 conn(0x7f8a2c06afc0 msgr2=0x7f8a2c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- 172.21.15.32:0/2872378005 shutdown_connections 2024-03-16T20:10:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- 172.21.15.32:0/2872378005 wait complete. 2024-03-16T20:10:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 Processor -- start 2024-03-16T20:10:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- start start 2024-03-16T20:10:02.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a2c072c70 0x7f8a2c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a2c073050 0x7f8a2c1473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c3820 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c3a20 con 0x7f8a2c073050 2024-03-16T20:10:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.162+0000 7f8a31764640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c4b80 con 0x7f8a2c072c70 2024-03-16T20:10:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.166+0000 7f8a2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.166+0000 7f8a2b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37370/0 (socket says 172.21.15.32:37370) 2024-03-16T20:10:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.166+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/3483848164 learned_addr learned my addr 172.21.15.32:0/3483848164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.166+0000 7f8a2affd640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a2c072c70 0x7f8a2c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.166+0000 7f8a2a7fc640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a2c073050 0x7f8a2c1473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a2c072c70 msgr2=0x7f8a2c0c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a2c072c70 0x7f8a2c0c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a2c073050 msgr2=0x7f8a2c1473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a2c073050 0x7f8a2c1473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 -- 172.21.15.32:0/3483848164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24009a70 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.170+0000 7f8a2b7fe640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8a2401a570 tx=0x7f8a240196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:02.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240addc0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a240b9450 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c0caa20 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240b97e0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c0cafa0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.178+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0674e0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.454+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8a240b9940 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.458+0000 7f8a137fe640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 0x7f8a18069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.458+0000 7f8a2affd640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 0x7f8a18069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:02.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.458+0000 7f8a2affd640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 0x7f8a18069350 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c005890 tx=0x7f8a1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.482+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8a240f19a0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.482+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8a240edc40 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.678+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a2c06d2d0 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.702+0000 7f8a137fe640 1 -- 172.21.15.32:0/3483848164 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f8a240b2020 con 0x7f8a2c0c16a0 2024-03-16T20:10:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:02.696 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:02.696 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 msgr2=0x7f8a18069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 0x7f8a18069350 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c005890 tx=0x7f8a1c009040 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 msgr2=0x7f8a2c14e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8a2401a570 tx=0x7f8a240196e0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 shutdown_connections 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f8a18066e90 0x7f8a18069350 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a2c0c16a0 0x7f8a2c14e480 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a2c073050 0x7f8a2c1473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 --2- 172.21.15.32:0/3483848164 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a2c072c70 0x7f8a2c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 >> 172.21.15.32:0/3483848164 conn(0x7f8a2c06afc0 msgr2=0x7f8a2c1509f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 shutdown_connections 2024-03-16T20:10:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:02.706+0000 7f8a31764640 1 -- 172.21.15.32:0/3483848164 wait complete. 2024-03-16T20:10:03.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:03.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:04.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 Processor -- start 2024-03-16T20:10:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 -- start start 2024-03-16T20:10:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb31c0c8550 0x7fb31c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c15ab60 con 0x7fb31c1544a0 2024-03-16T20:10:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c06f910 con 0x7fb31c0c8550 2024-03-16T20:10:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb3243ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c159fb0 con 0x7fb31c1540c0 2024-03-16T20:10:04.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.434+0000 7fb322156640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:04.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.438+0000 7fb322156640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:52766/0 (socket says 172.21.15.32:52766) 2024-03-16T20:10:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.438+0000 7fb322156640 1 -- 172.21.15.32:0/3962495940 learned_addr learned my addr 172.21.15.32:0/3962495940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb321154640 1 -- 172.21.15.32:0/3962495940 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373168881 0 0) 0x7fb31c15ab60 con 0x7fb31c1544a0 2024-03-16T20:10:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb321154640 1 -- 172.21.15.32:0/3962495940 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb308003530 con 0x7fb31c1544a0 2024-03-16T20:10:04.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb322156640 1 -- 172.21.15.32:0/3962495940 >> v2:172.21.15.32:3300/0 conn(0x7fb31c0c8550 msgr2=0x7fb31c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:04.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb322156640 1 --2- 172.21.15.32:0/3962495940 >> v2:172.21.15.32:3300/0 conn(0x7fb31c0c8550 0x7fb31c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:04.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb322156640 1 -- 172.21.15.32:0/3962495940 >> v1:172.21.15.59:6789/0 conn(0x7fb31c1544a0 legacy=0x7fb31c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:04.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.442+0000 7fb322156640 1 -- 172.21.15.32:0/3962495940 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c06fb80 con 0x7fb31c1540c0 2024-03-16T20:10:04.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.450+0000 7fb322156640 1 --2- 172.21.15.32:0/3962495940 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb31801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67b76f6a590b2545 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:04.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.462+0000 7fb321154640 1 -- 172.21.15.32:0/3962495940 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180ab600 con 0x7fb31c1540c0 2024-03-16T20:10:04.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.462+0000 7fb321154640 1 -- 172.21.15.32:0/3962495940 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3180a7050 con 0x7fb31c1540c0 2024-03-16T20:10:04.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb321154640 1 -- 172.21.15.32:0/3962495940 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180ab980 con 0x7fb31c1540c0 2024-03-16T20:10:04.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- 172.21.15.32:0/3962495940 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 msgr2=0x7fb31c06f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 --2- 172.21.15.32:0/3962495940 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb31801c560 comp rx=0 tx=0).stop 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- 172.21.15.32:0/3962495940 shutdown_connections 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 --2- 172.21.15.32:0/3962495940 >> v2:172.21.15.32:3300/0 conn(0x7fb31c0c8550 0x7fb31c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 --2- 172.21.15.32:0/3962495940 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c1540c0 0x7fb31c06f3d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- 172.21.15.32:0/3962495940 >> 172.21.15.32:0/3962495940 conn(0x7fb31c066ea0 msgr2=0x7fb31c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- 172.21.15.32:0/3962495940 shutdown_connections 2024-03-16T20:10:04.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- 172.21.15.32:0/3962495940 wait complete. 2024-03-16T20:10:04.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 Processor -- start 2024-03-16T20:10:04.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.466+0000 7fb3243ba640 1 -- start start 2024-03-16T20:10:04.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c0c8550 0x7fb31c1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb31c1540c0 0x7fb31c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c14e7c0 con 0x7fb31c1544a0 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0cc9f0 con 0x7fb31c1540c0 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb3243ba640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c14cce0 con 0x7fb31c0c8550 2024-03-16T20:10:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb321955640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb31c1540c0 0x7fb31c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:04.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb322957640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:04.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb322957640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37386/0 (socket says 172.21.15.32:37386) 2024-03-16T20:10:04.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb322957640 1 -- 172.21.15.32:0/1474163443 learned_addr learned my addr 172.21.15.32:0/1474163443 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:04.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.470+0000 7fb322156640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c0c8550 0x7fb31c1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:04.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.474+0000 7fb322957640 1 -- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c0c8550 msgr2=0x7fb31c1f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:04.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.474+0000 7fb322957640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c0c8550 0x7fb31c1f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:04.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.474+0000 7fb322957640 1 -- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb31c1540c0 msgr2=0x7fb31c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:04.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.474+0000 7fb322957640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb31c1540c0 0x7fb31c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:04.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.474+0000 7fb322957640 1 -- 172.21.15.32:0/1474163443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180094d0 con 0x7fb31c1544a0 2024-03-16T20:10:04.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.478+0000 7fb322957640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb31400b9e0 tx=0x7fb31400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400cb20 con 0x7fb31c1544a0 2024-03-16T20:10:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31400cca0 con 0x7fb31c1544a0 2024-03-16T20:10:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0c7760 con 0x7fb31c1544a0 2024-03-16T20:10:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314014680 con 0x7fb31c1544a0 2024-03-16T20:10:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c0cc680 con 0x7fb31c1544a0 2024-03-16T20:10:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.482+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c0cc840 con 0x7fb31c1544a0 2024-03-16T20:10:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.514+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb31401d3b0 con 0x7fb31c1544a0 2024-03-16T20:10:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.514+0000 7fb312ffd640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 0x7fb300069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.514+0000 7fb322156640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 0x7fb300069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.514+0000 7fb322156640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 0x7fb300069350 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0ce300 tx=0x7fb3180a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:04.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.518+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb314055990 con 0x7fb31c1544a0 2024-03-16T20:10:04.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:04.806+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb314051c30 con 0x7fb31c1544a0 2024-03-16T20:10:05.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.014+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31c0ccb60 con 0x7fb31c1544a0 2024-03-16T20:10:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.094+0000 7fb312ffd640 1 -- 172.21.15.32:0/1474163443 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb314019020 con 0x7fb31c1544a0 2024-03-16T20:10:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:05.088 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 msgr2=0x7fb300069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 0x7fb300069350 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0ce300 tx=0x7fb3180a6000 comp rx=0 tx=0).stop 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 msgr2=0x7fb31c0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb31400b9e0 tx=0x7fb31400bea0 comp rx=0 tx=0).stop 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 shutdown_connections 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb300066e90 0x7fb300069350 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb31c1544a0 0x7fb31c0cb660 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb31c1540c0 0x7fb31c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 --2- 172.21.15.32:0/1474163443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb31c0c8550 0x7fb31c1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 >> 172.21.15.32:0/1474163443 conn(0x7fb31c066ea0 msgr2=0x7fb31c150150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 shutdown_connections 2024-03-16T20:10:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:05.098+0000 7fb3243ba640 1 -- 172.21.15.32:0/1474163443 wait complete. 2024-03-16T20:10:05.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:05.712 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:06.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 Processor -- start 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 -- start start 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d540c7470 0x7f5d54165780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54070c90 con 0x7f5d54153fa0 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540c8590 con 0x7f5d54070840 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d5b497640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540c3f00 con 0x7f5d540c7470 2024-03-16T20:10:06.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d59a34640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d540c7470 0x7f5d54165780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:06.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d59a34640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d540c7470 0x7f5d54165780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:58358/0 (socket says 172.21.15.32:58358) 2024-03-16T20:10:06.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.830+0000 7f5d59a34640 1 -- 172.21.15.32:0/1154264311 learned_addr learned my addr 172.21.15.32:0/1154264311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:06.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.834+0000 7f5d59a34640 1 -- 172.21.15.32:0/1154264311 >> v1:172.21.15.59:6789/0 conn(0x7f5d54070840 legacy=0x7f5d540724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:06.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.834+0000 7f5d59a34640 1 -- 172.21.15.32:0/1154264311 >> v1:172.21.15.32:6789/0 conn(0x7f5d54153fa0 legacy=0x7f5d5415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:06.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.834+0000 7f5d59a34640 1 -- 172.21.15.32:0/1154264311 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54073590 con 0x7f5d540c7470 2024-03-16T20:10:06.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.842+0000 7f5d59a34640 1 --2- 172.21.15.32:0/1154264311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d540c7470 0x7f5d54165780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009e50 tx=0x7f5d4c01f1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feeda2af4af4acdc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d4bfff640 1 -- 172.21.15.32:0/1154264311 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0af400 con 0x7f5d540c7470 2024-03-16T20:10:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d4bfff640 1 -- 172.21.15.32:0/1154264311 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c0ab050 con 0x7f5d540c7470 2024-03-16T20:10:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d4bfff640 1 -- 172.21.15.32:0/1154264311 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0af780 con 0x7f5d540c7470 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- 172.21.15.32:0/1154264311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d540c7470 msgr2=0x7f5d54165780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 --2- 172.21.15.32:0/1154264311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d540c7470 0x7f5d54165780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009e50 tx=0x7f5d4c01f1a0 comp rx=0 tx=0).stop 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- 172.21.15.32:0/1154264311 shutdown_connections 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 --2- 172.21.15.32:0/1154264311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d540c7470 0x7f5d54165780 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- 172.21.15.32:0/1154264311 >> 172.21.15.32:0/1154264311 conn(0x7f5d5406c0d0 msgr2=0x7f5d5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- 172.21.15.32:0/1154264311 shutdown_connections 2024-03-16T20:10:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- 172.21.15.32:0/1154264311 wait complete. 2024-03-16T20:10:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 Processor -- start 2024-03-16T20:10:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.850+0000 7f5d5b497640 1 -- start start 2024-03-16T20:10:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d54070840 0x7f5d5413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5d540c7470 0x7f5d5413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54141660 con 0x7f5d54153fa0 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54146920 con 0x7f5d54070840 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d5b497640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54152a10 con 0x7f5d540c7470 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d59a34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d59a34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37396/0 (socket says 172.21.15.32:37396) 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d59a34640 1 -- 172.21.15.32:0/739952152 learned_addr learned my addr 172.21.15.32:0/739952152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d58a32640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5d540c7470 0x7f5d5413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:06.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.854+0000 7f5d59233640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d54070840 0x7f5d5413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.858+0000 7f5d59a34640 1 -- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5d540c7470 msgr2=0x7f5d5413fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.858+0000 7f5d59a34640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5d540c7470 0x7f5d5413fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.858+0000 7f5d59a34640 1 -- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d54070840 msgr2=0x7f5d5413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.858+0000 7f5d59a34640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d54070840 0x7f5d5413f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.858+0000 7f5d59a34640 1 -- 172.21.15.32:0/739952152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c009b30 con 0x7f5d54153fa0 2024-03-16T20:10:06.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.862+0000 7f5d59a34640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c01d850 tx=0x7f5d4c01bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0afd40 con 0x7f5d54153fa0 2024-03-16T20:10:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c0b9410 con 0x7f5d54153fa0 2024-03-16T20:10:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d54152cf0 con 0x7f5d54153fa0 2024-03-16T20:10:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0afea0 con 0x7f5d54153fa0 2024-03-16T20:10:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54141840 con 0x7f5d54153fa0 2024-03-16T20:10:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:06.866+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d541531a0 con 0x7f5d54153fa0 2024-03-16T20:10:07.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.182+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5d4c0b6050 con 0x7f5d54153fa0 2024-03-16T20:10:07.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.182+0000 7f5d49ffb640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 0x7f5d38069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:07.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.186+0000 7f5d59233640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 0x7f5d38069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:07.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.186+0000 7f5d59233640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 0x7f5d38069350 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007fd0 tx=0x7f5d50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.210+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f5d4c0f2f10 con 0x7f5d54153fa0 2024-03-16T20:10:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.210+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f5d4c0f2ce0 con 0x7f5d54153fa0 2024-03-16T20:10:07.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.466+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d54070c70 con 0x7f5d54153fa0 2024-03-16T20:10:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.490+0000 7f5d49ffb640 1 -- 172.21.15.32:0/739952152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f5d4c0b4030 con 0x7f5d54153fa0 2024-03-16T20:10:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:07.484 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:07.485 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 msgr2=0x7f5d38069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 0x7f5d38069350 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007fd0 tx=0x7f5d50009040 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 msgr2=0x7f5d54141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c01d850 tx=0x7f5d4c01bf40 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 shutdown_connections 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5d38066e90 0x7f5d38069350 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54153fa0 0x7f5d54141d60 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5d540c7470 0x7f5d5413fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 --2- 172.21.15.32:0/739952152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d54070840 0x7f5d5413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 >> 172.21.15.32:0/739952152 conn(0x7f5d5406c0d0 msgr2=0x7f5d54149d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 shutdown_connections 2024-03-16T20:10:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:07.494+0000 7f5d5b497640 1 -- 172.21.15.32:0/739952152 wait complete. 2024-03-16T20:10:08.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:08.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:09.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:09.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 Processor -- start 2024-03-16T20:10:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 -- start start 2024-03-16T20:10:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7a880725e0 0x7f7a88073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a88072dd0 con 0x7f7a8815a420 2024-03-16T20:10:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8815d790 con 0x7f7a88150d70 2024-03-16T20:10:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.226+0000 7f7a90557640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8815da50 con 0x7f7a880725e0 2024-03-16T20:10:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.230+0000 7f7a8eaf4640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a8815a420 0x7f7a8815a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43414/0 (socket says 172.21.15.32:43414) 2024-03-16T20:10:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.230+0000 7f7a8eaf4640 1 -- 172.21.15.32:0/2441252020 learned_addr learned my addr 172.21.15.32:0/2441252020 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.230+0000 7f7a8daf2640 1 --2- 172.21.15.32:0/2441252020 >> v2:172.21.15.59:3300/0 conn(0x7f7a880725e0 0x7f7a88073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.234+0000 7f7a8d2f1640 1 -- 172.21.15.32:0/2441252020 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944781285 0 0) 0x7f7a88072dd0 con 0x7f7a8815a420 2024-03-16T20:10:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.234+0000 7f7a8d2f1640 1 -- 172.21.15.32:0/2441252020 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a74003530 con 0x7f7a8815a420 2024-03-16T20:10:09.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.238+0000 7f7a8daf2640 1 -- 172.21.15.32:0/2441252020 >> v1:172.21.15.59:6789/0 conn(0x7f7a88150d70 legacy=0x7f7a88070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.238+0000 7f7a8daf2640 1 -- 172.21.15.32:0/2441252020 >> v1:172.21.15.32:6789/0 conn(0x7f7a8815a420 legacy=0x7f7a8815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.238+0000 7f7a8daf2640 1 -- 172.21.15.32:0/2441252020 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8815dd30 con 0x7f7a880725e0 2024-03-16T20:10:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.246+0000 7f7a8daf2640 1 --2- 172.21.15.32:0/2441252020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a880725e0 0x7f7a88073070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7a780097c0 tx=0x7f7a7801c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e38b2dfbfecb567 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a8d2f1640 1 -- 172.21.15.32:0/2441252020 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78004230 con 0x7f7a880725e0 2024-03-16T20:10:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a8d2f1640 1 -- 172.21.15.32:0/2441252020 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a780a7050 con 0x7f7a880725e0 2024-03-16T20:10:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a8d2f1640 1 -- 172.21.15.32:0/2441252020 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78003620 con 0x7f7a880725e0 2024-03-16T20:10:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- 172.21.15.32:0/2441252020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a880725e0 msgr2=0x7f7a88073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 --2- 172.21.15.32:0/2441252020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a880725e0 0x7f7a88073070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7a780097c0 tx=0x7f7a7801c350 comp rx=0 tx=0).stop 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- 172.21.15.32:0/2441252020 shutdown_connections 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 --2- 172.21.15.32:0/2441252020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a880725e0 0x7f7a88073070 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- 172.21.15.32:0/2441252020 >> 172.21.15.32:0/2441252020 conn(0x7f7a8806c0d0 msgr2=0x7f7a8806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- 172.21.15.32:0/2441252020 shutdown_connections 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- 172.21.15.32:0/2441252020 wait complete. 2024-03-16T20:10:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 Processor -- start 2024-03-16T20:10:09.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.254+0000 7f7a90557640 1 -- start start 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a88150d70 0x7f7a881f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a8815a420 0x7f7a88147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881f2350 con 0x7f7a880725e0 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881f6340 con 0x7f7a88150d70 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a90557640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881f25f0 con 0x7f7a8815a420 2024-03-16T20:10:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a8e2f3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a8eaf4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a8815a420 0x7f7a88147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a8e2f3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39248/0 (socket says 172.21.15.32:39248) 2024-03-16T20:10:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.258+0000 7f7a8e2f3640 1 -- 172.21.15.32:0/233977825 learned_addr learned my addr 172.21.15.32:0/233977825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8daf2640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a88150d70 0x7f7a881f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 -- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a8815a420 msgr2=0x7f7a88147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a8815a420 0x7f7a88147830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 -- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a88150d70 msgr2=0x7f7a881f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a88150d70 0x7f7a881f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 -- 172.21.15.32:0/233977825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78009420 con 0x7f7a880725e0 2024-03-16T20:10:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.262+0000 7f7a8e2f3640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c3c0 tx=0x7f7a8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.266+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840052d0 con 0x7f7a880725e0 2024-03-16T20:10:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.270+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84005430 con 0x7f7a880725e0 2024-03-16T20:10:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.270+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8815d560 con 0x7f7a880725e0 2024-03-16T20:10:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.270+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84003840 con 0x7f7a880725e0 2024-03-16T20:10:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.270+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8814fbe0 con 0x7f7a880725e0 2024-03-16T20:10:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.270+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88068000 con 0x7f7a880725e0 2024-03-16T20:10:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.594+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a840039a0 con 0x7f7a880725e0 2024-03-16T20:10:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.594+0000 7f7a7effd640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 0x7f7a6c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.594+0000 7f7a8daf2640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 0x7f7a6c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.594+0000 7f7a8daf2640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 0x7f7a6c069380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ad000 tx=0x7f7a780a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:09.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.610+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7a84054f40 con 0x7f7a880725e0 2024-03-16T20:10:09.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.610+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7a840511e0 con 0x7f7a880725e0 2024-03-16T20:10:09.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.818+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a880722b0 con 0x7f7a880725e0 2024-03-16T20:10:09.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.870+0000 7f7a7effd640 1 -- 172.21.15.32:0/233977825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f7a84017020 con 0x7f7a880725e0 2024-03-16T20:10:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:09.865 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:09.865 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:09.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 msgr2=0x7f7a6c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 0x7f7a6c069380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ad000 tx=0x7f7a780a6000 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 msgr2=0x7f7a88149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c3c0 tx=0x7f7a8400c880 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 shutdown_connections 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7a6c066ec0 0x7f7a6c069380 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7a8815a420 0x7f7a88147830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a88150d70 0x7f7a881f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 --2- 172.21.15.32:0/233977825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880725e0 0x7f7a88149380 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 >> 172.21.15.32:0/233977825 conn(0x7f7a8806c0d0 msgr2=0x7f7a8806f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 shutdown_connections 2024-03-16T20:10:09.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:09.874+0000 7f7a90557640 1 -- 172.21.15.32:0/233977825 wait complete. 2024-03-16T20:10:10.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:10.492 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:11.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:11.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.606+0000 7f5230e8d640 1 Processor -- start 2024-03-16T20:10:11.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.606+0000 7f5230e8d640 1 -- start start 2024-03-16T20:10:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f5230e8d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f522c150e60 0x7f522c15ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f5230e8d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c15a140 con 0x7f522c0c59c0 2024-03-16T20:10:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f5230e8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c15a700 con 0x7f522c072ac0 2024-03-16T20:10:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f5230e8d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c0c4e30 con 0x7f522c150e60 2024-03-16T20:10:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f522a59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f522c150e60 0x7f522c15ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f522a59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f522c150e60 0x7f522c15ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39256/0 (socket says 172.21.15.32:39256) 2024-03-16T20:10:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.610+0000 7f522a59c640 1 -- 172.21.15.32:0/2083342389 learned_addr learned my addr 172.21.15.32:0/2083342389 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.614+0000 7f522a59c640 1 -- 172.21.15.32:0/2083342389 >> v1:172.21.15.59:6789/0 conn(0x7f522c072ac0 legacy=0x7f522c0cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.614+0000 7f522a59c640 1 -- 172.21.15.32:0/2083342389 >> v1:172.21.15.32:6789/0 conn(0x7f522c0c59c0 legacy=0x7f522c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.614+0000 7f522a59c640 1 -- 172.21.15.32:0/2083342389 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c15b0d0 con 0x7f522c150e60 2024-03-16T20:10:11.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.618+0000 7f522a59c640 1 --2- 172.21.15.32:0/2083342389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c150e60 0x7f522c15ab90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f521c009870 tx=0x7f521c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ac39491d1c4140 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:11.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.622+0000 7f522959a640 1 -- 172.21.15.32:0/2083342389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0ab600 con 0x7f522c150e60 2024-03-16T20:10:11.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.622+0000 7f522959a640 1 -- 172.21.15.32:0/2083342389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0ab760 con 0x7f522c150e60 2024-03-16T20:10:11.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.622+0000 7f522959a640 1 -- 172.21.15.32:0/2083342389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0abaf0 con 0x7f522c150e60 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- 172.21.15.32:0/2083342389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c150e60 msgr2=0x7f522c15ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 --2- 172.21.15.32:0/2083342389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c150e60 0x7f522c15ab90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f521c009870 tx=0x7f521c01c4f0 comp rx=0 tx=0).stop 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- 172.21.15.32:0/2083342389 shutdown_connections 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 --2- 172.21.15.32:0/2083342389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c150e60 0x7f522c15ab90 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- 172.21.15.32:0/2083342389 >> 172.21.15.32:0/2083342389 conn(0x7f522c0673e0 msgr2=0x7f522c06c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- 172.21.15.32:0/2083342389 shutdown_connections 2024-03-16T20:10:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- 172.21.15.32:0/2083342389 wait complete. 2024-03-16T20:10:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 Processor -- start 2024-03-16T20:10:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- start start 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f522c150e60 0x7f522c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c154010 0x7f522c1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1f3c90 con 0x7f522c0c59c0 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1f3f50 con 0x7f522c154010 2024-03-16T20:10:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f5230e8d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1f4230 con 0x7f522c150e60 2024-03-16T20:10:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f522a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f522ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c154010 0x7f522c1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f522a59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39260/0 (socket says 172.21.15.32:39260) 2024-03-16T20:10:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.626+0000 7f522a59c640 1 -- 172.21.15.32:0/3375209937 learned_addr learned my addr 172.21.15.32:0/3375209937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.630+0000 7f5229d9b640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f522c150e60 0x7f522c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.634+0000 7f522a59c640 1 -- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f522c150e60 msgr2=0x7f522c0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.634+0000 7f522a59c640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f522c150e60 0x7f522c0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.634+0000 7f522a59c640 1 -- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c154010 msgr2=0x7f522c1f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.634+0000 7f522a59c640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c154010 0x7f522c1f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.634+0000 7f522a59c640 1 -- 172.21.15.32:0/3375209937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0094d0 con 0x7f522c0c59c0 2024-03-16T20:10:11.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.638+0000 7f522a59c640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f521c0ad040 tx=0x7f521c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:11.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.642+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0b4800 con 0x7f522c0c59c0 2024-03-16T20:10:11.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.642+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0b4960 con 0x7f522c0c59c0 2024-03-16T20:10:11.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.642+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1f4510 con 0x7f522c0c59c0 2024-03-16T20:10:11.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.642+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0b4cf0 con 0x7f522c0c59c0 2024-03-16T20:10:11.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.642+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c1f4ad0 con 0x7f522c0c59c0 2024-03-16T20:10:11.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.646+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c1f1b90 con 0x7f522c0c59c0 2024-03-16T20:10:11.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.678+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f521c0c7390 con 0x7f522c0c59c0 2024-03-16T20:10:11.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.678+0000 7f521affd640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 0x7f520c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:11.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.682+0000 7f5229d9b640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 0x7f520c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:11.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.682+0000 7f5229d9b640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 0x7f520c069300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f522c0c8e40 tx=0x7f5220007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.714+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f521c0f00c0 con 0x7f522c0c59c0 2024-03-16T20:10:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:11.982+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f521c0ec360 con 0x7f522c0c59c0 2024-03-16T20:10:12.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.346+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f522c06a340 con 0x7f522c0c59c0 2024-03-16T20:10:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f521affd640 1 -- 172.21.15.32:0/3375209937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f521c0efa20 con 0x7f522c0c59c0 2024-03-16T20:10:12.374 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:12.375 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 msgr2=0x7f520c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 0x7f520c069300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f522c0c8e40 tx=0x7f5220007e40 comp rx=0 tx=0).stop 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 msgr2=0x7f522c15b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f521c0ad040 tx=0x7f521c01ad30 comp rx=0 tx=0).stop 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 shutdown_connections 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f520c066e40 0x7f520c069300 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c154010 0x7f522c1f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f522c150e60 0x7f522c0c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 --2- 172.21.15.32:0/3375209937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f522c0c59c0 0x7f522c15b320 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.382+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 >> 172.21.15.32:0/3375209937 conn(0x7f522c072ac0 msgr2=0x7f522c0ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.386+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 shutdown_connections 2024-03-16T20:10:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:12.386+0000 7f5230e8d640 1 -- 172.21.15.32:0/3375209937 wait complete. 2024-03-16T20:10:12.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:10:12.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:10:12.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:12.997 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:13.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 Processor -- start 2024-03-16T20:10:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 -- start start 2024-03-16T20:10:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efe901542c0 0x7efe90073010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe90070360 0x7efe90072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe90072a70 0x7efe9015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe901546a0 con 0x7efe90070360 2024-03-16T20:10:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe90073550 con 0x7efe901542c0 2024-03-16T20:10:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe97ee9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe90159f20 con 0x7efe90072a70 2024-03-16T20:10:14.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe95484640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe90070360 0x7efe90072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe95484640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe90070360 0x7efe90072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39270/0 (socket says 172.21.15.32:39270) 2024-03-16T20:10:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.114+0000 7efe95484640 1 -- 172.21.15.32:0/3637289383 learned_addr learned my addr 172.21.15.32:0/3637289383 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95c85640 1 --2- 172.21.15.32:0/3637289383 >> v2:172.21.15.59:3300/0 conn(0x7efe901542c0 0x7efe90073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe96486640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe90072a70 0x7efe9015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95484640 1 -- 172.21.15.32:0/3637289383 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe90072a70 msgr2=0x7efe9015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95484640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe90072a70 0x7efe9015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95484640 1 -- 172.21.15.32:0/3637289383 >> v2:172.21.15.59:3300/0 conn(0x7efe901542c0 msgr2=0x7efe90073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95484640 1 --2- 172.21.15.32:0/3637289383 >> v2:172.21.15.59:3300/0 conn(0x7efe901542c0 0x7efe90073010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.118+0000 7efe95484640 1 -- 172.21.15.32:0/3637289383 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9015a1e0 con 0x7efe90070360 2024-03-16T20:10:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.122+0000 7efe95484640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90070360 0x7efe90072530 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7efe900c9130 tx=0x7efe8001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da6ee9bbe5b8fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:14.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe94c83640 1 -- 172.21.15.32:0/3637289383 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800ab600 con 0x7efe90070360 2024-03-16T20:10:14.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe94c83640 1 -- 172.21.15.32:0/3637289383 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe800a7050 con 0x7efe90070360 2024-03-16T20:10:14.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe94c83640 1 -- 172.21.15.32:0/3637289383 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800ab980 con 0x7efe90070360 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 -- 172.21.15.32:0/3637289383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90070360 msgr2=0x7efe90072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90070360 0x7efe90072530 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7efe900c9130 tx=0x7efe8001c520 comp rx=0 tx=0).stop 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 -- 172.21.15.32:0/3637289383 shutdown_connections 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe90072a70 0x7efe9015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 --2- 172.21.15.32:0/3637289383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90070360 0x7efe90072530 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 --2- 172.21.15.32:0/3637289383 >> v2:172.21.15.59:3300/0 conn(0x7efe901542c0 0x7efe90073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 -- 172.21.15.32:0/3637289383 >> 172.21.15.32:0/3637289383 conn(0x7efe9006c0d0 msgr2=0x7efe9006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 -- 172.21.15.32:0/3637289383 shutdown_connections 2024-03-16T20:10:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.126+0000 7efe97ee9640 1 -- 172.21.15.32:0/3637289383 wait complete. 2024-03-16T20:10:14.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 Processor -- start 2024-03-16T20:10:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 -- start start 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe90070360 0x7efe901f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe901542c0 0x7efe90140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe901f3760 con 0x7efe90072a70 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90148ea0 con 0x7efe90070360 2024-03-16T20:10:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe97ee9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe90149cb0 con 0x7efe901542c0 2024-03-16T20:10:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe95484640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe95484640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39286/0 (socket says 172.21.15.32:39286) 2024-03-16T20:10:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe95484640 1 -- 172.21.15.32:0/2439256517 learned_addr learned my addr 172.21.15.32:0/2439256517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe96486640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe901542c0 0x7efe90140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.130+0000 7efe95c85640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe90070360 0x7efe901f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.134+0000 7efe95484640 1 -- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe901542c0 msgr2=0x7efe90140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.134+0000 7efe95484640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe901542c0 0x7efe90140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.134+0000 7efe95484640 1 -- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe90070360 msgr2=0x7efe901f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.134+0000 7efe95484640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe90070360 0x7efe901f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.134+0000 7efe95484640 1 -- 172.21.15.32:0/2439256517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800094d0 con 0x7efe90072a70 2024-03-16T20:10:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.138+0000 7efe95484640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efe8001abd0 tx=0x7efe800192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.142+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800b4420 con 0x7efe90072a70 2024-03-16T20:10:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.142+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe800b4580 con 0x7efe90072a70 2024-03-16T20:10:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.142+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe900c6c60 con 0x7efe90072a70 2024-03-16T20:10:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.142+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800b4930 con 0x7efe90072a70 2024-03-16T20:10:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.142+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe901499d0 con 0x7efe90072a70 2024-03-16T20:10:14.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.146+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe90068000 con 0x7efe90072a70 2024-03-16T20:10:14.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.166+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efe800b2070 con 0x7efe90072a70 2024-03-16T20:10:14.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.166+0000 7efe867fc640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 0x7efe74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:14.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.166+0000 7efe95c85640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 0x7efe74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:14.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.170+0000 7efe95c85640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 0x7efe74069300 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009b60 tx=0x7efe8c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:14.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.486+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efe800eea60 con 0x7efe90072a70 2024-03-16T20:10:14.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.486+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efe800ead00 con 0x7efe90072a70 2024-03-16T20:10:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.694+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe90072310 con 0x7efe90072a70 2024-03-16T20:10:14.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.706+0000 7efe867fc640 1 -- 172.21.15.32:0/2439256517 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7efe800ba020 con 0x7efe90072a70 2024-03-16T20:10:14.701 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:14.701 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 msgr2=0x7efe74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 0x7efe74069300 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009b60 tx=0x7efe8c009190 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 msgr2=0x7efe9015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efe8001abd0 tx=0x7efe800192c0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 shutdown_connections 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efe74066e40 0x7efe74069300 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efe901542c0 0x7efe90140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe90072a70 0x7efe9015b340 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 --2- 172.21.15.32:0/2439256517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe90070360 0x7efe901f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 >> 172.21.15.32:0/2439256517 conn(0x7efe9006c0d0 msgr2=0x7efe900c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 shutdown_connections 2024-03-16T20:10:14.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:14.710+0000 7efe97ee9640 1 -- 172.21.15.32:0/2439256517 wait complete. 2024-03-16T20:10:15.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:15.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:16.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 Processor -- start 2024-03-16T20:10:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 -- start start 2024-03-16T20:10:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d8150d70 0x7f06d8073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f06d80725e0 0x7f06d8070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d8072dd0 con 0x7f06d80708a0 2024-03-16T20:10:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d815d790 con 0x7f06d80725e0 2024-03-16T20:10:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06dca9c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d815da50 con 0x7f06d8150d70 2024-03-16T20:10:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06d5d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f06d80725e0 0x7f06d8070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06d659c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d8150d70 0x7f06d8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06d5d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f06d80725e0 0x7f06d8070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:58608/0 (socket says 172.21.15.32:58608) 2024-03-16T20:10:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.450+0000 7f06d5d9b640 1 -- 172.21.15.32:0/2469878956 learned_addr learned my addr 172.21.15.32:0/2469878956 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.454+0000 7f06d5d9b640 1 -- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d8150d70 msgr2=0x7f06d8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.454+0000 7f06d5d9b640 1 --2- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d8150d70 0x7f06d8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.454+0000 7f06d5d9b640 1 -- 172.21.15.32:0/2469878956 >> v1:172.21.15.59:6789/0 conn(0x7f06d80708a0 legacy=0x7f06d815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.454+0000 7f06d5d9b640 1 -- 172.21.15.32:0/2469878956 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d815dd30 con 0x7f06d80725e0 2024-03-16T20:10:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.462+0000 7f06d5d9b640 1 --2- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d8070360 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009870 tx=0x7f06cc01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120e019062d9b971 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:16.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06d559a640 1 -- 172.21.15.32:0/2469878956 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0ab650 con 0x7f06d80725e0 2024-03-16T20:10:16.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06d559a640 1 -- 172.21.15.32:0/2469878956 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc0ab7b0 con 0x7f06d80725e0 2024-03-16T20:10:16.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06d559a640 1 -- 172.21.15.32:0/2469878956 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0abb40 con 0x7f06d80725e0 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 -- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 msgr2=0x7f06d8070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 --2- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d8070360 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009870 tx=0x7f06cc01c490 comp rx=0 tx=0).stop 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 -- 172.21.15.32:0/2469878956 shutdown_connections 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 --2- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d8070360 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 --2- 172.21.15.32:0/2469878956 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d8150d70 0x7f06d8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 -- 172.21.15.32:0/2469878956 >> 172.21.15.32:0/2469878956 conn(0x7f06d806c0d0 msgr2=0x7f06d806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 -- 172.21.15.32:0/2469878956 shutdown_connections 2024-03-16T20:10:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.466+0000 7f06dca9c640 1 -- 172.21.15.32:0/2469878956 wait complete. 2024-03-16T20:10:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 Processor -- start 2024-03-16T20:10:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 -- start start 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d80708a0 0x7f06d815c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d81f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d8152c00 con 0x7f06d8150d70 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80c64f0 con 0x7f06d80725e0 2024-03-16T20:10:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06dca9c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80c5ee0 con 0x7f06d80708a0 2024-03-16T20:10:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06d659c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d80708a0 0x7f06d815c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06d6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06d6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39300/0 (socket says 172.21.15.32:39300) 2024-03-16T20:10:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.470+0000 7f06d6d9d640 1 -- 172.21.15.32:0/2189260275 learned_addr learned my addr 172.21.15.32:0/2189260275 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:16.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.474+0000 7f06d5d9b640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d81f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.478+0000 7f06d6d9d640 1 -- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d80708a0 msgr2=0x7f06d815c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.478+0000 7f06d6d9d640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d80708a0 0x7f06d815c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.478+0000 7f06d6d9d640 1 -- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 msgr2=0x7f06d81f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.478+0000 7f06d6d9d640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d81f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.478+0000 7f06d6d9d640 1 -- 172.21.15.32:0/2189260275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0094d0 con 0x7f06d8150d70 2024-03-16T20:10:16.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.482+0000 7f06d6d9d640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f06d000c780 tx=0x7f06d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:16.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d0004b70 con 0x7f06d8150d70 2024-03-16T20:10:16.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06dca9c640 1 -- 172.21.15.32:0/2189260275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d815d560 con 0x7f06d8150d70 2024-03-16T20:10:16.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06dca9c640 1 -- 172.21.15.32:0/2189260275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d814a220 con 0x7f06d8150d70 2024-03-16T20:10:16.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06dca9c640 1 -- 172.21.15.32:0/2189260275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d8068000 con 0x7f06d8150d70 2024-03-16T20:10:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d000ce90 con 0x7f06d8150d70 2024-03-16T20:10:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.486+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d0006b60 con 0x7f06d8150d70 2024-03-16T20:10:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.838+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f06d000a7f0 con 0x7f06d8150d70 2024-03-16T20:10:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.838+0000 7f06c6ffd640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 0x7f06b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.838+0000 7f06d659c640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 0x7f06b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:16.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.838+0000 7f06d659c640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 0x7f06b8069300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009ac0 tx=0x7f06c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.870+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f06d0055af0 con 0x7f06d8150d70 2024-03-16T20:10:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:16.870+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f06d0051d90 con 0x7f06d8150d70 2024-03-16T20:10:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.094+0000 7f06dca9c640 1 -- 172.21.15.32:0/2189260275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d80c6630 con 0x7f06d8150d70 2024-03-16T20:10:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c6ffd640 1 -- 172.21.15.32:0/2189260275 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f06d0055450 con 0x7f06d8150d70 2024-03-16T20:10:17.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:17.160 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:17.160 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 msgr2=0x7f06b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 0x7f06b8069300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009ac0 tx=0x7f06c8009190 comp rx=0 tx=0).stop 2024-03-16T20:10:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 msgr2=0x7f06d81511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f06d000c780 tx=0x7f06d000cc40 comp rx=0 tx=0).stop 2024-03-16T20:10:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 shutdown_connections 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f06b8066e40 0x7f06b8069300 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d8150d70 0x7f06d81511e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06d80725e0 0x7f06d81f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 --2- 172.21.15.32:0/2189260275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06d80708a0 0x7f06d815c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 >> 172.21.15.32:0/2189260275 conn(0x7f06d806c0d0 msgr2=0x7f06d814e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 shutdown_connections 2024-03-16T20:10:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:17.166+0000 7f06c4ff9640 1 -- 172.21.15.32:0/2189260275 wait complete. 2024-03-16T20:10:17.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:17.781 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:18.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 Processor -- start 2024-03-16T20:10:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 -- start start 2024-03-16T20:10:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 0x7fc960073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc96015a420 0x7fc96015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc960151020 con 0x7fc960150c40 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96015a000 con 0x7fc96015a420 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.898+0000 7fc966f66640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96015adf0 con 0x7fc9600725e0 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.902+0000 7fc964d02640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc960150c40 0x7fc960070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43424/0 (socket says 172.21.15.32:43424) 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.902+0000 7fc964d02640 1 -- 172.21.15.32:0/1750442545 learned_addr learned my addr 172.21.15.32:0/1750442545 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.902+0000 7fc95ffff640 1 --2- 172.21.15.32:0/1750442545 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 0x7fc960073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:18.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.902+0000 7fc965503640 1 --2- 172.21.15.32:0/1750442545 >> v2:172.21.15.59:3300/0 conn(0x7fc96015a420 0x7fc96015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.906+0000 7fc95f7fe640 1 -- 172.21.15.32:0/1750442545 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685938198 0 0) 0x7fc960151020 con 0x7fc960150c40 2024-03-16T20:10:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.906+0000 7fc95f7fe640 1 -- 172.21.15.32:0/1750442545 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950003530 con 0x7fc960150c40 2024-03-16T20:10:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.910+0000 7fc95ffff640 1 -- 172.21.15.32:0/1750442545 >> v2:172.21.15.59:3300/0 conn(0x7fc96015a420 msgr2=0x7fc96015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.910+0000 7fc95ffff640 1 --2- 172.21.15.32:0/1750442545 >> v2:172.21.15.59:3300/0 conn(0x7fc96015a420 0x7fc96015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.910+0000 7fc95ffff640 1 -- 172.21.15.32:0/1750442545 >> v1:172.21.15.32:6789/0 conn(0x7fc960150c40 legacy=0x7fc960070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.910+0000 7fc95ffff640 1 -- 172.21.15.32:0/1750442545 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc960072da0 con 0x7fc9600725e0 2024-03-16T20:10:18.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.914+0000 7fc95ffff640 1 --2- 172.21.15.32:0/1750442545 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 0x7fc960073070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0097c0 tx=0x7fc94c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b765470bd8fe96 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc95f7fe640 1 -- 172.21.15.32:0/1750442545 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0a6040 con 0x7fc9600725e0 2024-03-16T20:10:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc95f7fe640 1 -- 172.21.15.32:0/1750442545 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c0aa600 con 0x7fc9600725e0 2024-03-16T20:10:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc95f7fe640 1 -- 172.21.15.32:0/1750442545 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0aa990 con 0x7fc9600725e0 2024-03-16T20:10:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- 172.21.15.32:0/1750442545 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 msgr2=0x7fc960073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- 172.21.15.32:0/1750442545 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 0x7fc960073070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0097c0 tx=0x7fc94c01c4a0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- 172.21.15.32:0/1750442545 shutdown_connections 2024-03-16T20:10:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- 172.21.15.32:0/1750442545 >> v2:172.21.15.59:3300/0 conn(0x7fc96015a420 0x7fc96015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- 172.21.15.32:0/1750442545 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc9600725e0 0x7fc960073070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- 172.21.15.32:0/1750442545 >> 172.21.15.32:0/1750442545 conn(0x7fc96006c0d0 msgr2=0x7fc96006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- 172.21.15.32:0/1750442545 shutdown_connections 2024-03-16T20:10:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- 172.21.15.32:0/1750442545 wait complete. 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 Processor -- start 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- start start 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960150c40 0x7fc9601f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96015a420 0x7fc9601f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f6cb0 con 0x7fc9600725e0 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600c71a0 con 0x7fc960150c40 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc966f66640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f6030 con 0x7fc96015a420 2024-03-16T20:10:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.926+0000 7fc964d02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39316/0 (socket says 172.21.15.32:39316) 2024-03-16T20:10:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 -- 172.21.15.32:0/4100365974 learned_addr learned my addr 172.21.15.32:0/4100365974 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc95ffff640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960150c40 0x7fc9601f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc965503640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96015a420 0x7fc9601f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 -- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96015a420 msgr2=0x7fc9601f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96015a420 0x7fc9601f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 -- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960150c40 msgr2=0x7fc9601f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960150c40 0x7fc9601f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.930+0000 7fc964d02640 1 -- 172.21.15.32:0/4100365974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c009420 con 0x7fc9600725e0 2024-03-16T20:10:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.934+0000 7fc964d02640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc9540010f0 tx=0x7fc95400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954005cf0 con 0x7fc9600725e0 2024-03-16T20:10:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc954005e70 con 0x7fc9600725e0 2024-03-16T20:10:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9540072f0 con 0x7fc9600725e0 2024-03-16T20:10:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9601f6310 con 0x7fc9600725e0 2024-03-16T20:10:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9601f68d0 con 0x7fc9600725e0 2024-03-16T20:10:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:18.942+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9601f6df0 con 0x7fc9600725e0 2024-03-16T20:10:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.262+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc954007450 con 0x7fc9600725e0 2024-03-16T20:10:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.262+0000 7fc95d7fa640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 0x7fc944069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.262+0000 7fc95ffff640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 0x7fc944069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.262+0000 7fc95ffff640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 0x7fc944069300 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0ae000 tx=0x7fc94c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.282+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc954054ea0 con 0x7fc9600725e0 2024-03-16T20:10:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.282+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc954051140 con 0x7fc9600725e0 2024-03-16T20:10:19.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.506+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc960068000 con 0x7fc9600725e0 2024-03-16T20:10:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc95d7fa640 1 -- 172.21.15.32:0/4100365974 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc954017020 con 0x7fc9600725e0 2024-03-16T20:10:19.531 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:19.531 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 msgr2=0x7fc944069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 0x7fc944069300 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0ae000 tx=0x7fc94c0b5040 comp rx=0 tx=0).stop 2024-03-16T20:10:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 msgr2=0x7fc96015c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc9540010f0 tx=0x7fc95400c890 comp rx=0 tx=0).stop 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 shutdown_connections 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc944066e40 0x7fc944069300 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc96015a420 0x7fc9601f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc960150c40 0x7fc9601f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 --2- 172.21.15.32:0/4100365974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9600725e0 0x7fc96015c470 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 >> 172.21.15.32:0/4100365974 conn(0x7fc96006c0d0 msgr2=0x7fc9600700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 shutdown_connections 2024-03-16T20:10:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:19.538+0000 7fc966f66640 1 -- 172.21.15.32:0/4100365974 wait complete. 2024-03-16T20:10:20.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:20.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:21.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:21.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 Processor -- start 2024-03-16T20:10:21.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 -- start start 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7848154030 0x7f7848072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78480cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784806b3f0 con 0x7f78480c9de0 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784806b990 con 0x7f7848154030 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f784d98a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784806b6b0 con 0x7f7848154410 2024-03-16T20:10:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f7846ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7848154030 0x7f7848072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f78467fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f78480c9de0 0x7f7848159ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:47232/0 (socket says 172.21.15.32:47232) 2024-03-16T20:10:21.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.269+0000 7f78467fc640 1 -- 172.21.15.32:0/1166422721 learned_addr learned my addr 172.21.15.32:0/1166422721 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:21.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.273+0000 7f78477fe640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78480cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.273+0000 7f7846ffd640 1 -- 172.21.15.32:0/1166422721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 msgr2=0x7f78480cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.273+0000 7f7846ffd640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78480cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.273+0000 7f7846ffd640 1 -- 172.21.15.32:0/1166422721 >> v1:172.21.15.59:6789/0 conn(0x7f78480c9de0 legacy=0x7f7848159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.273+0000 7f7846ffd640 1 -- 172.21.15.32:0/1166422721 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78480c9300 con 0x7f7848154030 2024-03-16T20:10:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.277+0000 7f7846ffd640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f7848072ec0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f78380097c0 tx=0x7f783801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433dd9fc166f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f7845ffb640 1 -- 172.21.15.32:0/1166422721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380a8990 con 0x7f7848154030 2024-03-16T20:10:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f7845ffb640 1 -- 172.21.15.32:0/1166422721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78380a8af0 con 0x7f7848154030 2024-03-16T20:10:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f7845ffb640 1 -- 172.21.15.32:0/1166422721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380a8e80 con 0x7f7848154030 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- 172.21.15.32:0/1166422721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 msgr2=0x7f7848072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f7848072ec0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f78380097c0 tx=0x7f783801ec90 comp rx=0 tx=0).stop 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- 172.21.15.32:0/1166422721 shutdown_connections 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78480cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- 172.21.15.32:0/1166422721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f7848072ec0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- 172.21.15.32:0/1166422721 >> 172.21.15.32:0/1166422721 conn(0x7f784806e6c0 msgr2=0x7f784806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- 172.21.15.32:0/1166422721 shutdown_connections 2024-03-16T20:10:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- 172.21.15.32:0/1166422721 wait complete. 2024-03-16T20:10:21.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 Processor -- start 2024-03-16T20:10:21.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- start start 2024-03-16T20:10:21.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78480c9de0 0x7f78480c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78481565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78481469a0 con 0x7f7848154030 2024-03-16T20:10:21.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784814a340 con 0x7f78480c9de0 2024-03-16T20:10:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.285+0000 7f784d98a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78481463e0 con 0x7f7848154410 2024-03-16T20:10:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.289+0000 7f78467fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.289+0000 7f78467fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39674/0 (socket says 172.21.15.32:39674) 2024-03-16T20:10:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.289+0000 7f78467fc640 1 -- 172.21.15.32:0/1128184827 learned_addr learned my addr 172.21.15.32:0/1128184827 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.289+0000 7f78477fe640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78481565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.289+0000 7f7846ffd640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78480c9de0 0x7f78480c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.293+0000 7f78467fc640 1 -- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 msgr2=0x7f78481565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.293+0000 7f78467fc640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78481565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.293+0000 7f78467fc640 1 -- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78480c9de0 msgr2=0x7f78480c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.293+0000 7f78467fc640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78480c9de0 0x7f78480c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.293+0000 7f78467fc640 1 -- 172.21.15.32:0/1128184827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009420 con 0x7f7848154030 2024-03-16T20:10:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.297+0000 7f78467fc640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f783c00c420 tx=0x7f783c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c005cf0 con 0x7f7848154030 2024-03-16T20:10:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c005e50 con 0x7f7848154030 2024-03-16T20:10:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f784d98a640 1 -- 172.21.15.32:0/1128184827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78481466c0 con 0x7f7848154030 2024-03-16T20:10:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c0072f0 con 0x7f7848154030 2024-03-16T20:10:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f784d98a640 1 -- 172.21.15.32:0/1128184827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78480c50d0 con 0x7f7848154030 2024-03-16T20:10:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.301+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78200052a0 con 0x7f7848154030 2024-03-16T20:10:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.337+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f783c007450 con 0x7f7848154030 2024-03-16T20:10:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.337+0000 7f782ffff640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 0x7f7834069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.337+0000 7f7846ffd640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 0x7f7834069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.337+0000 7f7846ffd640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 0x7f7834069350 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f78380b2000 tx=0x7f7838008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:21.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.617+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f783c055000 con 0x7f7848154030 2024-03-16T20:10:21.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.617+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f783c0512a0 con 0x7f7848154030 2024-03-16T20:10:21.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.833+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7820005900 con 0x7f7848154030 2024-03-16T20:10:21.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.933+0000 7f782ffff640 1 -- 172.21.15.32:0/1128184827 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f7820005900 con 0x7f7848154030 2024-03-16T20:10:21.929 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:21.930 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:21.930 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 msgr2=0x7f7834069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 0x7f7834069350 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f78380b2000 tx=0x7f7838008040 comp rx=0 tx=0).stop 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 msgr2=0x7f78480c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f783c00c420 tx=0x7f783c00c8e0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 shutdown_connections 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7834066e90 0x7f7834069350 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7848154410 0x7f78481565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7848154030 0x7f78480c5e10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 --2- 172.21.15.32:0/1128184827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78480c9de0 0x7f78480c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 >> 172.21.15.32:0/1128184827 conn(0x7f784806e6c0 msgr2=0x7f784806c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 shutdown_connections 2024-03-16T20:10:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:21.937+0000 7f782dffb640 1 -- 172.21.15.32:0/1128184827 wait complete. 2024-03-16T20:10:22.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:22.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:23.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.673+0000 7fc0a8f9f640 1 Processor -- start 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.673+0000 7fc0a8f9f640 1 -- start start 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a8f9f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 0x7fc0a40c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a8f9f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0a4158390 0x7fc0a41516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a8f9f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a4070760 con 0x7fc0a40c6040 2024-03-16T20:10:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a8f9f640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a40731c0 con 0x7fc0a4158390 2024-03-16T20:10:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a8f9f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40734a0 con 0x7fc0a40c6420 2024-03-16T20:10:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a2d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0a4158390 0x7fc0a41516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a259c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0a40c6040 0x7fc0a4157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39946/0 (socket says 172.21.15.32:39946) 2024-03-16T20:10:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a259c640 1 -- 172.21.15.32:0/1825999562 learned_addr learned my addr 172.21.15.32:0/1825999562 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:23.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.677+0000 7fc0a1d9b640 1 --2- 172.21.15.32:0/1825999562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 0x7fc0a40c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.685+0000 7fc0a1d9b640 1 -- 172.21.15.32:0/1825999562 >> v2:172.21.15.32:3300/0 conn(0x7fc0a4158390 msgr2=0x7fc0a41516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.685+0000 7fc0a1d9b640 1 --2- 172.21.15.32:0/1825999562 >> v2:172.21.15.32:3300/0 conn(0x7fc0a4158390 0x7fc0a41516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.685+0000 7fc0a1d9b640 1 -- 172.21.15.32:0/1825999562 >> v1:172.21.15.32:6789/0 conn(0x7fc0a40c6040 legacy=0x7fc0a4157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.685+0000 7fc0a1d9b640 1 -- 172.21.15.32:0/1825999562 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a406fe30 con 0x7fc0a40c6420 2024-03-16T20:10:23.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.689+0000 7fc0a1d9b640 1 --2- 172.21.15.32:0/1825999562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 0x7fc0a40c8800 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0980097f0 tx=0x7fc09801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b6504f29cbd2b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.693+0000 7fc0a159a640 1 -- 172.21.15.32:0/1825999562 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0980ab670 con 0x7fc0a40c6420 2024-03-16T20:10:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.693+0000 7fc0a159a640 1 -- 172.21.15.32:0/1825999562 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0980ab7d0 con 0x7fc0a40c6420 2024-03-16T20:10:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.693+0000 7fc0a159a640 1 -- 172.21.15.32:0/1825999562 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0980abb60 con 0x7fc0a40c6420 2024-03-16T20:10:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/1825999562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 msgr2=0x7fc0a40c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/1825999562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 0x7fc0a40c8800 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0980097f0 tx=0x7fc09801c470 comp rx=0 tx=0).stop 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/1825999562 shutdown_connections 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/1825999562 >> v2:172.21.15.32:3300/0 conn(0x7fc0a4158390 0x7fc0a41516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/1825999562 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6420 0x7fc0a40c8800 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/1825999562 >> 172.21.15.32:0/1825999562 conn(0x7fc0a406c0d0 msgr2=0x7fc0a406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/1825999562 shutdown_connections 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/1825999562 wait complete. 2024-03-16T20:10:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 Processor -- start 2024-03-16T20:10:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- start start 2024-03-16T20:10:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6040 0x7fc0a41400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a40c6420 0x7fc0a4140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a41412d0 con 0x7fc0a4158390 2024-03-16T20:10:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a4141980 con 0x7fc0a40c6420 2024-03-16T20:10:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a8f9f640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40c5790 con 0x7fc0a40c6040 2024-03-16T20:10:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.697+0000 7fc0a1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a40c6420 0x7fc0a4140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.701+0000 7fc0a2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39700/0 (socket says 172.21.15.32:39700) 2024-03-16T20:10:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.701+0000 7fc0a2d9d640 1 -- 172.21.15.32:0/2279212467 learned_addr learned my addr 172.21.15.32:0/2279212467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.701+0000 7fc0a259c640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6040 0x7fc0a41400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.705+0000 7fc0a2d9d640 1 -- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6040 msgr2=0x7fc0a41400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.705+0000 7fc0a2d9d640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6040 0x7fc0a41400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.705+0000 7fc0a2d9d640 1 -- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a40c6420 msgr2=0x7fc0a4140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.705+0000 7fc0a2d9d640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a40c6420 0x7fc0a4140d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.705+0000 7fc0a2d9d640 1 -- 172.21.15.32:0/2279212467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098009420 con 0x7fc0a4158390 2024-03-16T20:10:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.709+0000 7fc0a2d9d640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00efc0 tx=0x7fc09c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c009ba0 con 0x7fc0a4158390 2024-03-16T20:10:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c00f040 con 0x7fc0a4158390 2024-03-16T20:10:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a40c5a30 con 0x7fc0a4158390 2024-03-16T20:10:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0146c0 con 0x7fc0a4158390 2024-03-16T20:10:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a4158040 con 0x7fc0a4158390 2024-03-16T20:10:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.713+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a4068000 con 0x7fc0a4158390 2024-03-16T20:10:23.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.997+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc09c014820 con 0x7fc0a4158390 2024-03-16T20:10:23.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.997+0000 7fc092ffd640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 0x7fc084069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:23.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.997+0000 7fc0a259c640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 0x7fc084069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:23.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:23.997+0000 7fc0a259c640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 0x7fc084069300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc094000f70 tx=0x7fc094007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:24.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.025+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc09c056780 con 0x7fc0a4158390 2024-03-16T20:10:24.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.025+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc09c052a20 con 0x7fc0a4158390 2024-03-16T20:10:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.245+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0a40c7d40 con 0x7fc0a4158390 2024-03-16T20:10:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.265+0000 7fc092ffd640 1 -- 172.21.15.32:0/2279212467 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc09c018070 con 0x7fc0a4158390 2024-03-16T20:10:24.262 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:24.262 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 msgr2=0x7fc084069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 0x7fc084069300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc094000f70 tx=0x7fc094007e90 comp rx=0 tx=0).stop 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 msgr2=0x7fc0a413f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00efc0 tx=0x7fc09c00c490 comp rx=0 tx=0).stop 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 shutdown_connections 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc084066e40 0x7fc084069300 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a4158390 0x7fc0a413f320 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:24.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a40c6420 0x7fc0a4140d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 --2- 172.21.15.32:0/2279212467 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0a40c6040 0x7fc0a41400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 >> 172.21.15.32:0/2279212467 conn(0x7fc0a406c0d0 msgr2=0x7fc0a414fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 shutdown_connections 2024-03-16T20:10:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:24.269+0000 7fc0a8f9f640 1 -- 172.21.15.32:0/2279212467 wait complete. 2024-03-16T20:10:24.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:24.885 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:25.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 Processor -- start 2024-03-16T20:10:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 -- start start 2024-03-16T20:10:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf4c0c8720 0x7fdf4c0c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c06c460 con 0x7fdf4c15ac10 2024-03-16T20:10:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c15a240 con 0x7fdf4c0c8720 2024-03-16T20:10:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf50a49640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c06cf20 con 0x7fdf4c1540c0 2024-03-16T20:10:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf49d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf4ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf4c0c8720 0x7fdf4c0c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf49d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39712/0 (socket says 172.21.15.32:39712) 2024-03-16T20:10:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf49d9b640 1 -- 172.21.15.32:0/1879067574 learned_addr learned my addr 172.21.15.32:0/1879067574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.001+0000 7fdf4a59c640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.005+0000 7fdf49d9b640 1 -- 172.21.15.32:0/1879067574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 msgr2=0x7fdf4c15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.005+0000 7fdf49d9b640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c15a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.005+0000 7fdf49d9b640 1 -- 172.21.15.32:0/1879067574 >> v2:172.21.15.59:3300/0 conn(0x7fdf4c0c8720 msgr2=0x7fdf4c0c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.005+0000 7fdf49d9b640 1 --2- 172.21.15.32:0/1879067574 >> v2:172.21.15.59:3300/0 conn(0x7fdf4c0c8720 0x7fdf4c0c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.005+0000 7fdf49d9b640 1 -- 172.21.15.32:0/1879067574 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c06d200 con 0x7fdf4c15ac10 2024-03-16T20:10:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.009+0000 7fdf49d9b640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0cad40 tx=0x7fdf4001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fea2943cd06a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.013+0000 7fdf4959a640 1 -- 172.21.15.32:0/1879067574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf400ac620 con 0x7fdf4c15ac10 2024-03-16T20:10:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.013+0000 7fdf4959a640 1 -- 172.21.15.32:0/1879067574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf400ac780 con 0x7fdf4c15ac10 2024-03-16T20:10:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.013+0000 7fdf4959a640 1 -- 172.21.15.32:0/1879067574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf400acb10 con 0x7fdf4c15ac10 2024-03-16T20:10:26.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.013+0000 7fdf50a49640 1 -- 172.21.15.32:0/1879067574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 msgr2=0x7fdf4c0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:26.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.013+0000 7fdf50a49640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0cad40 tx=0x7fdf4001c4f0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- 172.21.15.32:0/1879067574 shutdown_connections 2024-03-16T20:10:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- 172.21.15.32:0/1879067574 >> v2:172.21.15.59:3300/0 conn(0x7fdf4c0c8720 0x7fdf4c0c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c0c81e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- 172.21.15.32:0/1879067574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- 172.21.15.32:0/1879067574 >> 172.21.15.32:0/1879067574 conn(0x7fdf4c070f90 msgr2=0x7fdf4c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- 172.21.15.32:0/1879067574 shutdown_connections 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- 172.21.15.32:0/1879067574 wait complete. 2024-03-16T20:10:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 Processor -- start 2024-03-16T20:10:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- start start 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf4c0c8720 0x7fdf4c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1561c0 con 0x7fdf4c15ac10 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1548a0 con 0x7fdf4c0c8720 2024-03-16T20:10:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.017+0000 7fdf50a49640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1f2f80 con 0x7fdf4c1540c0 2024-03-16T20:10:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.021+0000 7fdf4ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.021+0000 7fdf4a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf4c0c8720 0x7fdf4c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.021+0000 7fdf4ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39724/0 (socket says 172.21.15.32:39724) 2024-03-16T20:10:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.021+0000 7fdf49d9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.021+0000 7fdf4ad9d640 1 -- 172.21.15.32:0/2887140614 learned_addr learned my addr 172.21.15.32:0/2887140614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.025+0000 7fdf4ad9d640 1 -- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 msgr2=0x7fdf4c0ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.025+0000 7fdf4ad9d640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c0ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.025+0000 7fdf4ad9d640 1 -- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf4c0c8720 msgr2=0x7fdf4c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.025+0000 7fdf4ad9d640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf4c0c8720 0x7fdf4c0cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.025+0000 7fdf4ad9d640 1 -- 172.21.15.32:0/2887140614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400094d0 con 0x7fdf4c15ac10 2024-03-16T20:10:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.029+0000 7fdf4ad9d640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400eed0 tx=0x7fdf4400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44010040 con 0x7fdf4c15ac10 2024-03-16T20:10:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4400bd20 con 0x7fdf4c15ac10 2024-03-16T20:10:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf50a49640 1 -- 172.21.15.32:0/2887140614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c1f3220 con 0x7fdf4c15ac10 2024-03-16T20:10:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44014640 con 0x7fdf4c15ac10 2024-03-16T20:10:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf50a49640 1 -- 172.21.15.32:0/2887140614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c1f37e0 con 0x7fdf4c15ac10 2024-03-16T20:10:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.033+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf240052a0 con 0x7fdf4c15ac10 2024-03-16T20:10:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.329+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdf440147a0 con 0x7fdf4c15ac10 2024-03-16T20:10:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.329+0000 7fdf3affd640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 0x7fdf2c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.329+0000 7fdf4a59c640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 0x7fdf2c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.329+0000 7fdf4a59c640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 0x7fdf2c069330 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0079c0 tx=0x7fdf3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:26.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.353+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fdf44056380 con 0x7fdf4c15ac10 2024-03-16T20:10:26.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.353+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fdf44052620 con 0x7fdf4c15ac10 2024-03-16T20:10:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.557+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf24005900 con 0x7fdf4c15ac10 2024-03-16T20:10:26.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.573+0000 7fdf3affd640 1 -- 172.21.15.32:0/2887140614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fdf24005900 con 0x7fdf4c15ac10 2024-03-16T20:10:26.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:26.568 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 msgr2=0x7fdf2c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 0x7fdf2c069330 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0079c0 tx=0x7fdf3c008040 comp rx=0 tx=0).stop 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 msgr2=0x7fdf4c154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400eed0 tx=0x7fdf4400b480 comp rx=0 tx=0).stop 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 shutdown_connections 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fdf2c066e70 0x7fdf2c069330 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf4c15ac10 0x7fdf4c154d30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdf4c1540c0 0x7fdf4c0ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 --2- 172.21.15.32:0/2887140614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf4c0c8720 0x7fdf4c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:26.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 >> 172.21.15.32:0/2887140614 conn(0x7fdf4c070f90 msgr2=0x7fdf4c15d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:26.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 shutdown_connections 2024-03-16T20:10:26.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:26.577+0000 7fdf38ff9640 1 -- 172.21.15.32:0/2887140614 wait complete. 2024-03-16T20:10:27.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:27.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:28.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 Processor -- start 2024-03-16T20:10:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 -- start start 2024-03-16T20:10:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4930072c70 0x7f49300c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4930073050 0x7f49300c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493006e6a0 con 0x7f493006bae0 2024-03-16T20:10:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4930156f20 con 0x7f4930073050 2024-03-16T20:10:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f493675a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493006c250 con 0x7f4930072c70 2024-03-16T20:10:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f492f7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4930073050 0x7f49300c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f492f7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4930073050 0x7f49300c14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39726/0 (socket says 172.21.15.32:39726) 2024-03-16T20:10:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f492f7fe640 1 -- 172.21.15.32:0/3762345491 learned_addr learned my addr 172.21.15.32:0/3762345491 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.313+0000 7f492ffff640 1 --2- 172.21.15.32:0/3762345491 >> v2:172.21.15.59:3300/0 conn(0x7f4930072c70 0x7f49300c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.317+0000 7f492f7fe640 1 -- 172.21.15.32:0/3762345491 >> v2:172.21.15.59:3300/0 conn(0x7f4930072c70 msgr2=0x7f49300c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.317+0000 7f492f7fe640 1 --2- 172.21.15.32:0/3762345491 >> v2:172.21.15.59:3300/0 conn(0x7f4930072c70 0x7f49300c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.317+0000 7f492f7fe640 1 -- 172.21.15.32:0/3762345491 >> v1:172.21.15.32:6789/0 conn(0x7f493006bae0 legacy=0x7f4930150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.317+0000 7f492f7fe640 1 -- 172.21.15.32:0/3762345491 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493006c510 con 0x7f4930073050 2024-03-16T20:10:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.325+0000 7f492f7fe640 1 --2- 172.21.15.32:0/3762345491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930073050 0x7f49300c14d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f493006cfc0 tx=0x7f492401ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680feaa8b9afc131 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f492effd640 1 -- 172.21.15.32:0/3762345491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49240a7cb0 con 0x7f4930073050 2024-03-16T20:10:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f492effd640 1 -- 172.21.15.32:0/3762345491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49240a7e10 con 0x7f4930073050 2024-03-16T20:10:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f492effd640 1 -- 172.21.15.32:0/3762345491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49240af660 con 0x7f4930073050 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 -- 172.21.15.32:0/3762345491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930073050 msgr2=0x7f49300c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 --2- 172.21.15.32:0/3762345491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930073050 0x7f49300c14d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f493006cfc0 tx=0x7f492401ebc0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 -- 172.21.15.32:0/3762345491 shutdown_connections 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 --2- 172.21.15.32:0/3762345491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930073050 0x7f49300c14d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 --2- 172.21.15.32:0/3762345491 >> v2:172.21.15.59:3300/0 conn(0x7f4930072c70 0x7f49300c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 -- 172.21.15.32:0/3762345491 >> 172.21.15.32:0/3762345491 conn(0x7f493006afc0 msgr2=0x7f493006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 -- 172.21.15.32:0/3762345491 shutdown_connections 2024-03-16T20:10:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.329+0000 7f493675a640 1 -- 172.21.15.32:0/3762345491 wait complete. 2024-03-16T20:10:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 Processor -- start 2024-03-16T20:10:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 -- start start 2024-03-16T20:10:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493006bae0 0x7f4930144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4930073050 0x7f4930147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49301ef810 con 0x7f4930072c70 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49301eec20 con 0x7f493006bae0 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f493675a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49301472a0 con 0x7f4930073050 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f492f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f492f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39728/0 (socket says 172.21.15.32:39728) 2024-03-16T20:10:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f492f7fe640 1 -- 172.21.15.32:0/1865780225 learned_addr learned my addr 172.21.15.32:0/1865780225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f492ffff640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493006bae0 0x7f4930144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.333+0000 7f4934cf7640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4930073050 0x7f4930147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.337+0000 7f492f7fe640 1 -- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4930073050 msgr2=0x7f4930147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.337+0000 7f492f7fe640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4930073050 0x7f4930147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.337+0000 7f492f7fe640 1 -- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493006bae0 msgr2=0x7f4930144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.337+0000 7f492f7fe640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493006bae0 0x7f4930144830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.337+0000 7f492f7fe640 1 -- 172.21.15.32:0/1865780225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492400b770 con 0x7f4930072c70 2024-03-16T20:10:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.341+0000 7f492f7fe640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f492401b7d0 tx=0x7f492401b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.345+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49240afd10 con 0x7f4930072c70 2024-03-16T20:10:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.345+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49240afe90 con 0x7f4930072c70 2024-03-16T20:10:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.345+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49301eeee0 con 0x7f4930072c70 2024-03-16T20:10:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.345+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49240ba680 con 0x7f4930072c70 2024-03-16T20:10:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.345+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49301ef4a0 con 0x7f4930072c70 2024-03-16T20:10:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.349+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49301ef630 con 0x7f4930072c70 2024-03-16T20:10:28.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.369+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f49240b6040 con 0x7f4930072c70 2024-03-16T20:10:28.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.373+0000 7f492cff9640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 0x7f491c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:28.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.373+0000 7f492ffff640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 0x7f491c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:28.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.373+0000 7f492ffff640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 0x7f491c069350 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4920009b60 tx=0x7f4920009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.665+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f49240f2f50 con 0x7f4930072c70 2024-03-16T20:10:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.665+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f49240ef1f0 con 0x7f4930072c70 2024-03-16T20:10:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:28.937+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4930068000 con 0x7f4930072c70 2024-03-16T20:10:29.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.033+0000 7f492cff9640 1 -- 172.21.15.32:0/1865780225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f49240b4070 con 0x7f4930072c70 2024-03-16T20:10:29.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:29.028 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:29.029 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 msgr2=0x7f491c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 0x7f491c069350 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4920009b60 tx=0x7f4920009190 comp rx=0 tx=0).stop 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 msgr2=0x7f49301468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f492401b7d0 tx=0x7f492401b960 comp rx=0 tx=0).stop 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 shutdown_connections 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f491c066e90 0x7f491c069350 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4930073050 0x7f4930147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:29.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4930072c70 0x7f49301468f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:29.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 --2- 172.21.15.32:0/1865780225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493006bae0 0x7f4930144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:29.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 >> 172.21.15.32:0/1865780225 conn(0x7f493006afc0 msgr2=0x7f493006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:29.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 shutdown_connections 2024-03-16T20:10:29.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:29.037+0000 7f493675a640 1 -- 172.21.15.32:0/1865780225 wait complete. 2024-03-16T20:10:29.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:29.659 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:30.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:30.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 Processor -- start 2024-03-16T20:10:30.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 -- start start 2024-03-16T20:10:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb5500c6c40 0x7fb55006e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb550157ef0 con 0x7fb550071a50 2024-03-16T20:10:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55006eed0 con 0x7fb5500c7020 2024-03-16T20:10:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb556a5a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb550157c50 con 0x7fb5500c6c40 2024-03-16T20:10:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb554ff7640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb550071a50 0x7fb550151970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36146/0 (socket says 172.21.15.32:36146) 2024-03-16T20:10:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.777+0000 7fb554ff7640 1 -- 172.21.15.32:0/2447304377 learned_addr learned my addr 172.21.15.32:0/2447304377 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.781+0000 7fb54ffff640 1 --2- 172.21.15.32:0/2447304377 >> v2:172.21.15.59:3300/0 conn(0x7fb5500c6c40 0x7fb55006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.781+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314854228 0 0) 0x7fb550157ef0 con 0x7fb550071a50 2024-03-16T20:10:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.781+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb538003530 con 0x7fb550071a50 2024-03-16T20:10:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119229752 0 0) 0x7fb538003530 con 0x7fb550071a50 2024-03-16T20:10:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 >> v2:172.21.15.59:3300/0 conn(0x7fb5500c6c40 msgr2=0x7fb55006e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 --2- 172.21.15.32:0/2447304377 >> v2:172.21.15.59:3300/0 conn(0x7fb5500c6c40 0x7fb55006e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 >> v1:172.21.15.59:6789/0 conn(0x7fb5500c7020 legacy=0x7fb550158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550072b50 con 0x7fb550071a50 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7fb548001a00 con 0x7fb550071a50 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 >> v1:172.21.15.32:6789/0 conn(0x7fb550071a50 legacy=0x7fb550151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5380039d0 0x7fb538003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5380047d0 0x7fb538004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb550157c50 con 0x7fb538005190 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800c060 con 0x7fb5380047d0 2024-03-16T20:10:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800c220 con 0x7fb5380039d0 2024-03-16T20:10:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.785+0000 7fb54ffff640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5380039d0 0x7fb538003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.789+0000 7fb554ff7640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.793+0000 7fb554ff7640 1 -- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5380039d0 msgr2=0x7fb538003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.793+0000 7fb554ff7640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5380039d0 0x7fb538003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.793+0000 7fb554ff7640 1 -- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5380047d0 msgr2=0x7fb538004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.793+0000 7fb554ff7640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5380047d0 0x7fb538004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.793+0000 7fb554ff7640 1 -- 172.21.15.32:0/2447304377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538003530 con 0x7fb538005190 2024-03-16T20:10:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.797+0000 7fb554ff7640 1 -- 172.21.15.32:0/2447304377 reap_dead start 2024-03-16T20:10:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.797+0000 7fb554ff7640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 0x7fb53800b990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb54800a810 tx=0x7fb54801d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5230b9dbb13266da server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480ac620 con 0x7fb538005190 2024-03-16T20:10:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480a8070 con 0x7fb538005190 2024-03-16T20:10:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb54effd640 1 -- 172.21.15.32:0/2447304377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480aca00 con 0x7fb538005190 2024-03-16T20:10:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 -- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 msgr2=0x7fb53800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 0x7fb53800b990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb54800a810 tx=0x7fb54801d320 comp rx=0 tx=0).stop 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 -- 172.21.15.32:0/2447304377 shutdown_connections 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 --2- 172.21.15.32:0/2447304377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 -- 172.21.15.32:0/2447304377 >> 172.21.15.32:0/2447304377 conn(0x7fb5500cbfe0 msgr2=0x7fb5500cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 -- 172.21.15.32:0/2447304377 shutdown_connections 2024-03-16T20:10:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.801+0000 7fb556a5a640 1 -- 172.21.15.32:0/2447304377 wait complete. 2024-03-16T20:10:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 Processor -- start 2024-03-16T20:10:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 -- start start 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb538005190 0x7fb5501f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5501ed850 0x7fb55006b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501ecba0 con 0x7fb5501551b0 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500cbd80 con 0x7fb538005190 2024-03-16T20:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb556a5a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501ed3c0 con 0x7fb5501ed850 2024-03-16T20:10:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb54f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb54f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50186/0 (socket says 172.21.15.32:50186) 2024-03-16T20:10:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb54f7fe640 1 -- 172.21.15.32:0/6792091 learned_addr learned my addr 172.21.15.32:0/6792091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb554ff7640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5501ed850 0x7fb55006b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.805+0000 7fb54ffff640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb538005190 0x7fb5501f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 -- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5501ed850 msgr2=0x7fb55006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5501ed850 0x7fb55006b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 -- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb538005190 msgr2=0x7fb5501f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb538005190 0x7fb5501f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 -- 172.21.15.32:0/6792091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5501f06d0 con 0x7fb5501551b0 2024-03-16T20:10:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.813+0000 7fb54f7fe640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c3f0 tx=0x7fb54400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:30.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544011790 con 0x7fb5501551b0 2024-03-16T20:10:30.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440118f0 con 0x7fb5501551b0 2024-03-16T20:10:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5501ecdb0 con 0x7fb5501551b0 2024-03-16T20:10:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544011ca0 con 0x7fb5501551b0 2024-03-16T20:10:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5500cb7c0 con 0x7fb5501551b0 2024-03-16T20:10:30.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.821+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb550068000 con 0x7fb5501551b0 2024-03-16T20:10:30.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.893+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb544026430 con 0x7fb5501551b0 2024-03-16T20:10:30.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.893+0000 7fb54cff9640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 0x7fb53c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:30.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.893+0000 7fb54ffff640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 0x7fb53c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:30.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:30.893+0000 7fb54ffff640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 0x7fb53c069300 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb540009b60 tx=0x7fb540009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:31.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.217+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb5440564d0 con 0x7fb5501551b0 2024-03-16T20:10:31.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.217+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb544055ba0 con 0x7fb5501551b0 2024-03-16T20:10:31.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.477+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5500cbaa0 con 0x7fb5501551b0 2024-03-16T20:10:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb54cff9640 1 -- 172.21.15.32:0/6792091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb544003810 con 0x7fb5501551b0 2024-03-16T20:10:31.498 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:31.498 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 msgr2=0x7fb53c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 0x7fb53c069300 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb540009b60 tx=0x7fb540009190 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 msgr2=0x7fb550152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c3f0 tx=0x7fb54400c7f0 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 shutdown_connections 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb53c066e40 0x7fb53c069300 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb5501ed850 0x7fb55006b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5501551b0 0x7fb550152580 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 --2- 172.21.15.32:0/6792091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb538005190 0x7fb5501f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 >> 172.21.15.32:0/6792091 conn(0x7fb5500cbfe0 msgr2=0x7fb5500c6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 shutdown_connections 2024-03-16T20:10:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:31.505+0000 7fb556a5a640 1 -- 172.21.15.32:0/6792091 wait complete. 2024-03-16T20:10:32.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:32.119 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:33.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:33.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.233+0000 7ff9e4a02640 1 Processor -- start 2024-03-16T20:10:33.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.233+0000 7ff9e4a02640 1 -- start start 2024-03-16T20:10:33.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9e0150f90 0x7ff9e0156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e00c6fd0 0x7ff9e006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff9e006c610 0x7ff9e00c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00700f0 con 0x7ff9e0150f90 2024-03-16T20:10:33.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e0072e70 con 0x7ff9e006c610 2024-03-16T20:10:33.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9e4a02640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00728b0 con 0x7ff9e00c6fd0 2024-03-16T20:10:33.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9de59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9e0150f90 0x7ff9e0156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9ddd9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e00c6fd0 0x7ff9e006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9de59c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9e0150f90 0x7ff9e0156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50200/0 (socket says 172.21.15.32:50200) 2024-03-16T20:10:33.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9de59c640 1 -- 172.21.15.32:0/2490437092 learned_addr learned my addr 172.21.15.32:0/2490437092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:33.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.237+0000 7ff9ded9d640 1 --2- 172.21.15.32:0/2490437092 >> v2:172.21.15.59:3300/0 conn(0x7ff9e006c610 0x7ff9e00c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.241+0000 7ff9de59c640 1 -- 172.21.15.32:0/2490437092 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e00c6fd0 msgr2=0x7ff9e006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.241+0000 7ff9de59c640 1 --2- 172.21.15.32:0/2490437092 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e00c6fd0 0x7ff9e006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.241+0000 7ff9de59c640 1 -- 172.21.15.32:0/2490437092 >> v2:172.21.15.59:3300/0 conn(0x7ff9e006c610 msgr2=0x7ff9e00c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.241+0000 7ff9de59c640 1 --2- 172.21.15.32:0/2490437092 >> v2:172.21.15.59:3300/0 conn(0x7ff9e006c610 0x7ff9e00c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.241+0000 7ff9de59c640 1 -- 172.21.15.32:0/2490437092 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0072b90 con 0x7ff9e0150f90 2024-03-16T20:10:33.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.245+0000 7ff9de59c640 1 --2- 172.21.15.32:0/2490437092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e0150f90 0x7ff9e0156d30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00721d0 tx=0x7ff9d001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7990e66049ca4ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:33.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.249+0000 7ff9dd59a640 1 -- 172.21.15.32:0/2490437092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00ab5f0 con 0x7ff9e0150f90 2024-03-16T20:10:33.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.249+0000 7ff9dd59a640 1 -- 172.21.15.32:0/2490437092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d00ab750 con 0x7ff9e0150f90 2024-03-16T20:10:33.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.249+0000 7ff9dd59a640 1 -- 172.21.15.32:0/2490437092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00abae0 con 0x7ff9e0150f90 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- 172.21.15.32:0/2490437092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e0150f90 msgr2=0x7ff9e0156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/2490437092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e0150f90 0x7ff9e0156d30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00721d0 tx=0x7ff9d001c520 comp rx=0 tx=0).stop 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- 172.21.15.32:0/2490437092 shutdown_connections 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/2490437092 >> v2:172.21.15.59:3300/0 conn(0x7ff9e006c610 0x7ff9e00c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/2490437092 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e00c6fd0 0x7ff9e006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/2490437092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e0150f90 0x7ff9e0156d30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- 172.21.15.32:0/2490437092 >> 172.21.15.32:0/2490437092 conn(0x7ff9e00cc7a0 msgr2=0x7ff9e00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- 172.21.15.32:0/2490437092 shutdown_connections 2024-03-16T20:10:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- 172.21.15.32:0/2490437092 wait complete. 2024-03-16T20:10:33.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 Processor -- start 2024-03-16T20:10:33.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- start start 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e00c6fd0 0x7ff9e00cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e0150f90 0x7ff9e013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00cc030 con 0x7ff9e006c610 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0157f90 con 0x7ff9e00c6fd0 2024-03-16T20:10:33.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9e4a02640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0158270 con 0x7ff9e0150f90 2024-03-16T20:10:33.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9ded9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e0150f90 0x7ff9e013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50210/0 (socket says 172.21.15.32:50210) 2024-03-16T20:10:33.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9de59c640 1 -- 172.21.15.32:0/3712241385 learned_addr learned my addr 172.21.15.32:0/3712241385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:33.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.253+0000 7ff9ddd9b640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e00c6fd0 0x7ff9e00cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.257+0000 7ff9de59c640 1 -- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e0150f90 msgr2=0x7ff9e013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.257+0000 7ff9de59c640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e0150f90 0x7ff9e013ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.257+0000 7ff9de59c640 1 -- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e00c6fd0 msgr2=0x7ff9e00cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.257+0000 7ff9de59c640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e00c6fd0 0x7ff9e00cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.257+0000 7ff9de59c640 1 -- 172.21.15.32:0/3712241385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00094d0 con 0x7ff9e006c610 2024-03-16T20:10:33.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.261+0000 7ff9de59c640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff9d001abd0 tx=0x7ff9d001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00ab5f0 con 0x7ff9e006c610 2024-03-16T20:10:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d00be3f0 con 0x7ff9e006c610 2024-03-16T20:10:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0158550 con 0x7ff9e006c610 2024-03-16T20:10:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e013efd0 con 0x7ff9e006c610 2024-03-16T20:10:33.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00be7a0 con 0x7ff9e006c610 2024-03-16T20:10:33.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.265+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c40052a0 con 0x7ff9e006c610 2024-03-16T20:10:33.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.289+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff9d00b43f0 con 0x7ff9e006c610 2024-03-16T20:10:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.289+0000 7ff9ceffd640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 0x7ff9c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.289+0000 7ff9ddd9b640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 0x7ff9c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.293+0000 7ff9ddd9b640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 0x7ff9c0069300 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0154b00 tx=0x7ff9d4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.581+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff9d00a8050 con 0x7ff9e006c610 2024-03-16T20:10:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.581+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff9d00ebee0 con 0x7ff9e006c610 2024-03-16T20:10:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.833+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9c4005900 con 0x7ff9e006c610 2024-03-16T20:10:33.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9ceffd640 1 -- 172.21.15.32:0/3712241385 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7ff9c4005900 con 0x7ff9e006c610 2024-03-16T20:10:33.867 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:33.868 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:33.868 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 msgr2=0x7ff9c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 0x7ff9c0069300 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0154b00 tx=0x7ff9d4009190 comp rx=0 tx=0).stop 2024-03-16T20:10:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 msgr2=0x7ff9e015adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff9d001abd0 tx=0x7ff9d001ad60 comp rx=0 tx=0).stop 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 shutdown_connections 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff9c0066e40 0x7ff9c0069300 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff9e0150f90 0x7ff9e013ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9e00c6fd0 0x7ff9e00cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 --2- 172.21.15.32:0/3712241385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9e006c610 0x7ff9e015adc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 >> 172.21.15.32:0/3712241385 conn(0x7ff9e00cc7a0 msgr2=0x7ff9e014de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 shutdown_connections 2024-03-16T20:10:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:33.873+0000 7ff9e4a02640 1 -- 172.21.15.32:0/3712241385 wait complete. 2024-03-16T20:10:34.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:34.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:35.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.605+0000 7f87e4035640 1 Processor -- start 2024-03-16T20:10:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.605+0000 7f87e4035640 1 -- start start 2024-03-16T20:10:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e4035640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87dc151000 0x7f87dc0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e4035640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87dc0c8980 0x7f87dc15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e4035640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0c8ec0 con 0x7f87dc0c36b0 2024-03-16T20:10:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e4035640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc072f10 con 0x7f87dc151000 2024-03-16T20:10:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e4035640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc0c5000 con 0x7f87dc0c8980 2024-03-16T20:10:35.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87dc151000 0x7f87dc0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e25d2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87dc0c8980 0x7f87dc15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87dc151000 0x7f87dc0c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50212/0 (socket says 172.21.15.32:50212) 2024-03-16T20:10:35.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 -- 172.21.15.32:0/821334001 learned_addr learned my addr 172.21.15.32:0/821334001 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:35.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 -- 172.21.15.32:0/821334001 >> v2:172.21.15.59:3300/0 conn(0x7f87dc0c8980 msgr2=0x7f87dc15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:35.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 --2- 172.21.15.32:0/821334001 >> v2:172.21.15.59:3300/0 conn(0x7f87dc0c8980 0x7f87dc15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 -- 172.21.15.32:0/821334001 >> v1:172.21.15.59:6789/0 conn(0x7f87dc0c36b0 legacy=0x7f87dc0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:35.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.609+0000 7f87e1dd1640 1 -- 172.21.15.32:0/821334001 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc072950 con 0x7f87dc151000 2024-03-16T20:10:35.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.617+0000 7f87e1dd1640 1 --2- 172.21.15.32:0/821334001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc151000 0x7f87dc0c8420 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0725d0 tx=0x7f87d801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7469a101b6568a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:35.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e0dcf640 1 -- 172.21.15.32:0/821334001 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80aa640 con 0x7f87dc151000 2024-03-16T20:10:35.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e0dcf640 1 -- 172.21.15.32:0/821334001 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d80aa7a0 con 0x7f87dc151000 2024-03-16T20:10:35.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e0dcf640 1 -- 172.21.15.32:0/821334001 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80aab30 con 0x7f87dc151000 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 -- 172.21.15.32:0/821334001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc151000 msgr2=0x7f87dc0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 --2- 172.21.15.32:0/821334001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc151000 0x7f87dc0c8420 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0725d0 tx=0x7f87d801c4e0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 -- 172.21.15.32:0/821334001 shutdown_connections 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 --2- 172.21.15.32:0/821334001 >> v2:172.21.15.59:3300/0 conn(0x7f87dc0c8980 0x7f87dc15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 --2- 172.21.15.32:0/821334001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc151000 0x7f87dc0c8420 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 -- 172.21.15.32:0/821334001 >> 172.21.15.32:0/821334001 conn(0x7f87dc06c0d0 msgr2=0x7f87dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:35.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 -- 172.21.15.32:0/821334001 shutdown_connections 2024-03-16T20:10:35.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 -- 172.21.15.32:0/821334001 wait complete. 2024-03-16T20:10:35.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.621+0000 7f87e4035640 1 Processor -- start 2024-03-16T20:10:35.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 -- start start 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87dc0c8980 0x7f87dc14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f87dc162700 0x7f87dc146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc163660 con 0x7f87dc0c36b0 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1638b0 con 0x7f87dc0c8980 2024-03-16T20:10:35.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e4035640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc163b90 con 0x7f87dc162700 2024-03-16T20:10:35.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e1dd1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e15d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87dc0c8980 0x7f87dc14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e1dd1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50216/0 (socket says 172.21.15.32:50216) 2024-03-16T20:10:35.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e1dd1640 1 -- 172.21.15.32:0/2027503958 learned_addr learned my addr 172.21.15.32:0/2027503958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:35.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.625+0000 7f87e25d2640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f87dc162700 0x7f87dc146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.629+0000 7f87e1dd1640 1 -- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f87dc162700 msgr2=0x7f87dc146c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.629+0000 7f87e1dd1640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f87dc162700 0x7f87dc146c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.629+0000 7f87e1dd1640 1 -- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87dc0c8980 msgr2=0x7f87dc14d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.629+0000 7f87e1dd1640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87dc0c8980 0x7f87dc14d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.629+0000 7f87e1dd1640 1 -- 172.21.15.32:0/2027503958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80094d0 con 0x7f87dc0c36b0 2024-03-16T20:10:35.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.633+0000 7f87e1dd1640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f87d801ab90 tx=0x7f87d801ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.637+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80bd5f0 con 0x7f87dc0c36b0 2024-03-16T20:10:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.637+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d80bd750 con 0x7f87dc0c36b0 2024-03-16T20:10:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.637+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc14c430 con 0x7f87dc0c36b0 2024-03-16T20:10:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.641+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80bdb00 con 0x7f87dc0c36b0 2024-03-16T20:10:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.641+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc0c9490 con 0x7f87dc0c36b0 2024-03-16T20:10:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.641+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc068000 con 0x7f87dc0c36b0 2024-03-16T20:10:35.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.669+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f87d80a8050 con 0x7f87dc0c36b0 2024-03-16T20:10:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.669+0000 7f87d27fc640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 0x7f87c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.669+0000 7f87e15d0640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 0x7f87c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:35.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.673+0000 7f87e15d0640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 0x7f87c0069350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f87dc1f0ae0 tx=0x7f87cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.981+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f87d80aa5e0 con 0x7f87dc0c36b0 2024-03-16T20:10:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:35.981+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f87d80ec4f0 con 0x7f87dc0c36b0 2024-03-16T20:10:36.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.201+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87dc14c8e0 con 0x7f87dc0c36b0 2024-03-16T20:10:36.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.221+0000 7f87d27fc640 1 -- 172.21.15.32:0/2027503958 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f87d80efbb0 con 0x7f87dc0c36b0 2024-03-16T20:10:36.216 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:36.216 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:36.216 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 msgr2=0x7f87c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 0x7f87c0069350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f87dc1f0ae0 tx=0x7f87cc009040 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 msgr2=0x7f87dc14cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f87d801ab90 tx=0x7f87d801ad20 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 shutdown_connections 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f87c0066e90 0x7f87c0069350 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f87dc162700 0x7f87dc146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87dc0c8980 0x7f87dc14d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 --2- 172.21.15.32:0/2027503958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87dc0c36b0 0x7f87dc14cc30 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 >> 172.21.15.32:0/2027503958 conn(0x7f87dc06c0d0 msgr2=0x7f87dc0cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:36.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 shutdown_connections 2024-03-16T20:10:36.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:36.225+0000 7f87e4035640 1 -- 172.21.15.32:0/2027503958 wait complete. 2024-03-16T20:10:36.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:36.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:37.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 Processor -- start 2024-03-16T20:10:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 -- start start 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f780c6040 0x7f6f780c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f780c6420 0x7f6f78157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f7806ba30 0x7f6f781581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f78070760 con 0x7f6f7806ba30 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f780731c0 con 0x7f6f780c6040 2024-03-16T20:10:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f7e066640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780734a0 con 0x7f6f780c6420 2024-03-16T20:10:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f77fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f7806ba30 0x7f6f781581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f76ffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f780c6420 0x7f6f78157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f777fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f780c6040 0x7f6f780c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f77fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f7806ba30 0x7f6f781581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50228/0 (socket says 172.21.15.32:50228) 2024-03-16T20:10:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.969+0000 7f6f77fff640 1 -- 172.21.15.32:0/3074662019 learned_addr learned my addr 172.21.15.32:0/3074662019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.973+0000 7f6f77fff640 1 -- 172.21.15.32:0/3074662019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f780c6420 msgr2=0x7f6f78157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.973+0000 7f6f77fff640 1 --2- 172.21.15.32:0/3074662019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f780c6420 0x7f6f78157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.973+0000 7f6f77fff640 1 -- 172.21.15.32:0/3074662019 >> v2:172.21.15.59:3300/0 conn(0x7f6f780c6040 msgr2=0x7f6f780c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.973+0000 7f6f77fff640 1 --2- 172.21.15.32:0/3074662019 >> v2:172.21.15.59:3300/0 conn(0x7f6f780c6040 0x7f6f780c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.973+0000 7f6f77fff640 1 -- 172.21.15.32:0/3074662019 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7806fe30 con 0x7f6f7806ba30 2024-03-16T20:10:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.977+0000 7f6f77fff640 1 --2- 172.21.15.32:0/3074662019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f7806ba30 0x7f6f781581c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c3aa0 tx=0x7f6f7001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7959b873fe09c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f767fc640 1 -- 172.21.15.32:0/3074662019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f700a5870 con 0x7f6f7806ba30 2024-03-16T20:10:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f767fc640 1 -- 172.21.15.32:0/3074662019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f700a59d0 con 0x7f6f7806ba30 2024-03-16T20:10:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f767fc640 1 -- 172.21.15.32:0/3074662019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f700a5d60 con 0x7f6f7806ba30 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 -- 172.21.15.32:0/3074662019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f7806ba30 msgr2=0x7f6f781581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 --2- 172.21.15.32:0/3074662019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f7806ba30 0x7f6f781581c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c3aa0 tx=0x7f6f7001c9f0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 -- 172.21.15.32:0/3074662019 shutdown_connections 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 --2- 172.21.15.32:0/3074662019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f7806ba30 0x7f6f781581c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 --2- 172.21.15.32:0/3074662019 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f780c6420 0x7f6f78157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 --2- 172.21.15.32:0/3074662019 >> v2:172.21.15.59:3300/0 conn(0x7f6f780c6040 0x7f6f780c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 -- 172.21.15.32:0/3074662019 >> 172.21.15.32:0/3074662019 conn(0x7f6f7806c0d0 msgr2=0x7f6f7806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 -- 172.21.15.32:0/3074662019 shutdown_connections 2024-03-16T20:10:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.981+0000 7f6f7e066640 1 -- 172.21.15.32:0/3074662019 wait complete. 2024-03-16T20:10:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 Processor -- start 2024-03-16T20:10:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 -- start start 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f7806ba30 0x7f6f781f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f780c6420 0x7f6f780c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f781f5f30 con 0x7f6f780c6040 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f78148380 con 0x7f6f780c6420 2024-03-16T20:10:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f7e066640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f78150850 con 0x7f6f7806ba30 2024-03-16T20:10:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f76ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f777fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f7806ba30 0x7f6f781f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f76ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50238/0 (socket says 172.21.15.32:50238) 2024-03-16T20:10:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f76ffd640 1 -- 172.21.15.32:0/516492172 learned_addr learned my addr 172.21.15.32:0/516492172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.985+0000 7f6f77fff640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f780c6420 0x7f6f780c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.989+0000 7f6f76ffd640 1 -- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f7806ba30 msgr2=0x7f6f781f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.989+0000 7f6f76ffd640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f7806ba30 0x7f6f781f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.989+0000 7f6f76ffd640 1 -- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f780c6420 msgr2=0x7f6f780c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.989+0000 7f6f76ffd640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f780c6420 0x7f6f780c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.989+0000 7f6f76ffd640 1 -- 172.21.15.32:0/516492172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70009b30 con 0x7f6f780c6040 2024-03-16T20:10:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.993+0000 7f6f76ffd640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e840 tx=0x7f6f6c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.997+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c009510 con 0x7f6f780c6040 2024-03-16T20:10:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.997+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f78148640 con 0x7f6f780c6040 2024-03-16T20:10:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.997+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f781f58e0 con 0x7f6f780c6040 2024-03-16T20:10:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:37.997+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f781f60a0 con 0x7f6f780c6040 2024-03-16T20:10:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.205+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c00ee90 con 0x7f6f780c6040 2024-03-16T20:10:38.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.321+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c005ba0 con 0x7f6f780c6040 2024-03-16T20:10:38.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.321+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6f6c00c940 con 0x7f6f780c6040 2024-03-16T20:10:38.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.321+0000 7f6f5ffff640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 0x7f6f64069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:38.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.321+0000 7f6f777fe640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 0x7f6f64069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.321+0000 7f6f777fe640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 0x7f6f64069300 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6f68007a60 tx=0x7f6f68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.345+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6f6c055dc0 con 0x7f6f780c6040 2024-03-16T20:10:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.345+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6f6c052060 con 0x7f6f780c6040 2024-03-16T20:10:38.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.565+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f78068000 con 0x7f6f780c6040 2024-03-16T20:10:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.613+0000 7f6f5ffff640 1 -- 172.21.15.32:0/516492172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6f6c021070 con 0x7f6f780c6040 2024-03-16T20:10:38.608 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:38.609 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 msgr2=0x7f6f64069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 0x7f6f64069300 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6f68007a60 tx=0x7f6f68008040 comp rx=0 tx=0).stop 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 msgr2=0x7f6f7814faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e840 tx=0x7f6f6c00ec40 comp rx=0 tx=0).stop 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 shutdown_connections 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6f64066e40 0x7f6f64069300 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f780c6420 0x7f6f780c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f780c6040 0x7f6f7814faa0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 --2- 172.21.15.32:0/516492172 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6f7806ba30 0x7f6f781f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 >> 172.21.15.32:0/516492172 conn(0x7f6f7806c0d0 msgr2=0x7f6f781567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 shutdown_connections 2024-03-16T20:10:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:38.617+0000 7f6f7e066640 1 -- 172.21.15.32:0/516492172 wait complete. 2024-03-16T20:10:39.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:39.238 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:40.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:40.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.353+0000 7fc0262af640 1 Processor -- start 2024-03-16T20:10:40.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.353+0000 7fc0262af640 1 -- start start 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc0262af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc0200c6420 0x7fc0200c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc0262af640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc020158390 0x7fc0201516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc0262af640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc020070760 con 0x7fc0200c6040 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc0262af640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0200731c0 con 0x7fc0200c6420 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc0262af640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200734a0 con 0x7fc020158390 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc01ffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0200c6040 0x7fc020157c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41792/0 (socket says 172.21.15.32:41792) 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc01ffff640 1 -- 172.21.15.32:0/3661986886 learned_addr learned my addr 172.21.15.32:0/3661986886 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc01f7fe640 1 --2- 172.21.15.32:0/3661986886 >> v2:172.21.15.59:3300/0 conn(0x7fc0200c6420 0x7fc0200c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:40.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.357+0000 7fc02484c640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc020158390 0x7fc0201516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.361+0000 7fc01effd640 1 -- 172.21.15.32:0/3661986886 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802706296 0 0) 0x7fc020070760 con 0x7fc0200c6040 2024-03-16T20:10:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.361+0000 7fc01effd640 1 -- 172.21.15.32:0/3661986886 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc008003530 con 0x7fc0200c6040 2024-03-16T20:10:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.365+0000 7fc01f7fe640 1 -- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc020158390 msgr2=0x7fc0201516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.365+0000 7fc01f7fe640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc020158390 0x7fc0201516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.365+0000 7fc01f7fe640 1 -- 172.21.15.32:0/3661986886 >> v1:172.21.15.32:6789/0 conn(0x7fc0200c6040 legacy=0x7fc020157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.365+0000 7fc01f7fe640 1 -- 172.21.15.32:0/3661986886 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02006fe30 con 0x7fc0200c6420 2024-03-16T20:10:40.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.369+0000 7fc01f7fe640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6420 0x7fc0200c8800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc0140097c0 tx=0x7fc01401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f4a3afd67c1cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:40.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.377+0000 7fc01effd640 1 -- 172.21.15.32:0/3661986886 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0140ab670 con 0x7fc0200c6420 2024-03-16T20:10:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc01effd640 1 -- 172.21.15.32:0/3661986886 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0140ab7d0 con 0x7fc0200c6420 2024-03-16T20:10:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc01effd640 1 -- 172.21.15.32:0/3661986886 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0140abb40 con 0x7fc0200c6420 2024-03-16T20:10:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6420 msgr2=0x7fc0200c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6420 0x7fc0200c8800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc0140097c0 tx=0x7fc01401c470 comp rx=0 tx=0).stop 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- 172.21.15.32:0/3661986886 shutdown_connections 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc020158390 0x7fc0201516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- 172.21.15.32:0/3661986886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6420 0x7fc0200c8800 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- 172.21.15.32:0/3661986886 >> 172.21.15.32:0/3661986886 conn(0x7fc02006c0d0 msgr2=0x7fc02006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- 172.21.15.32:0/3661986886 shutdown_connections 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- 172.21.15.32:0/3661986886 wait complete. 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 Processor -- start 2024-03-16T20:10:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- start start 2024-03-16T20:10:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6040 0x7fc02015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0200c6420 0x7fc0201590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201f2ca0 con 0x7fc020158390 2024-03-16T20:10:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201f2f80 con 0x7fc0200c6040 2024-03-16T20:10:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.385+0000 7fc0262af640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201f3260 con 0x7fc0200c6420 2024-03-16T20:10:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.389+0000 7fc02484c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.389+0000 7fc02484c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42140/0 (socket says 172.21.15.32:42140) 2024-03-16T20:10:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.389+0000 7fc02484c640 1 -- 172.21.15.32:0/3967680318 learned_addr learned my addr 172.21.15.32:0/3967680318 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.389+0000 7fc01f7fe640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0200c6420 0x7fc0201590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 -- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0200c6420 msgr2=0x7fc0201590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0200c6420 0x7fc0201590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 -- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6040 msgr2=0x7fc02015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6040 0x7fc02015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 -- 172.21.15.32:0/3967680318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014009420 con 0x7fc020158390 2024-03-16T20:10:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.393+0000 7fc02484c640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc01800efc0 tx=0x7fc01800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018009ba0 con 0x7fc020158390 2024-03-16T20:10:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800f040 con 0x7fc020158390 2024-03-16T20:10:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0201f3540 con 0x7fc020158390 2024-03-16T20:10:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0180146c0 con 0x7fc020158390 2024-03-16T20:10:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc020153b60 con 0x7fc020158390 2024-03-16T20:10:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.401+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0200729b0 con 0x7fc020158390 2024-03-16T20:10:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.425+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc018014820 con 0x7fc020158390 2024-03-16T20:10:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.425+0000 7fc01cff9640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 0x7fc00c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.425+0000 7fc01ffff640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 0x7fc00c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.425+0000 7fc01ffff640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 0x7fc00c069350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc010007fd0 tx=0x7fc010007940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.717+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc018056e80 con 0x7fc020158390 2024-03-16T20:10:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:40.717+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fc018052a20 con 0x7fc020158390 2024-03-16T20:10:41.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.013+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc020068000 con 0x7fc020158390 2024-03-16T20:10:41.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.045+0000 7fc01cff9640 1 -- 172.21.15.32:0/3967680318 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fc018018030 con 0x7fc020158390 2024-03-16T20:10:41.038 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:41.038 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 msgr2=0x7fc00c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 0x7fc00c069350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc010007fd0 tx=0x7fc010007940 comp rx=0 tx=0).stop 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 msgr2=0x7fc020147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc01800efc0 tx=0x7fc01800c490 comp rx=0 tx=0).stop 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 shutdown_connections 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc00c066e90 0x7fc00c069350 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc020158390 0x7fc020147050 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0200c6420 0x7fc0201590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 --2- 172.21.15.32:0/3967680318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0200c6040 0x7fc02015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 >> 172.21.15.32:0/3967680318 conn(0x7fc02006c0d0 msgr2=0x7fc020156040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 shutdown_connections 2024-03-16T20:10:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:41.049+0000 7fc0262af640 1 -- 172.21.15.32:0/3967680318 wait complete. 2024-03-16T20:10:41.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:41.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:42.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.789+0000 7f26e991e640 1 Processor -- start 2024-03-16T20:10:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.789+0000 7f26e991e640 1 -- start start 2024-03-16T20:10:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e991e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26e40b1eb0 0x7f26e4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e991e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4072f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e991e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40c68e0 con 0x7f26e40c7ce0 2024-03-16T20:10:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e991e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e4067340 con 0x7f26e40b1eb0 2024-03-16T20:10:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e991e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e406c1f0 con 0x7f26e4158980 2024-03-16T20:10:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26e40b1eb0 0x7f26e4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26e40b1eb0 0x7f26e4072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:51082/0 (socket says 172.21.15.32:51082) 2024-03-16T20:10:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.793+0000 7f26e2ffd640 1 -- 172.21.15.32:0/2766577197 learned_addr learned my addr 172.21.15.32:0/2766577197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.801+0000 7f26e2ffd640 1 -- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 msgr2=0x7f26e4072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.801+0000 7f26e2ffd640 1 --2- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.801+0000 7f26e2ffd640 1 -- 172.21.15.32:0/2766577197 >> v1:172.21.15.59:6789/0 conn(0x7f26e40c7ce0 legacy=0x7f26e4158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.801+0000 7f26e2ffd640 1 -- 172.21.15.32:0/2766577197 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e406c4d0 con 0x7f26e40b1eb0 2024-03-16T20:10:42.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.805+0000 7f26e2ffd640 1 --2- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b1eb0 0x7f26e4072a10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009870 tx=0x7f26d401ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e140d85f4b7db3d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:42.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.809+0000 7f26e1ffb640 1 -- 172.21.15.32:0/2766577197 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d40a89d0 con 0x7f26e40b1eb0 2024-03-16T20:10:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.813+0000 7f26e1ffb640 1 -- 172.21.15.32:0/2766577197 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d40a8b30 con 0x7f26e40b1eb0 2024-03-16T20:10:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.813+0000 7f26e1ffb640 1 -- 172.21.15.32:0/2766577197 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d40b0440 con 0x7f26e40b1eb0 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b1eb0 msgr2=0x7f26e4072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b1eb0 0x7f26e4072a10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009870 tx=0x7f26d401ed10 comp rx=0 tx=0).stop 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- 172.21.15.32:0/2766577197 shutdown_connections 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- 172.21.15.32:0/2766577197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b1eb0 0x7f26e4072a10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- 172.21.15.32:0/2766577197 >> 172.21.15.32:0/2766577197 conn(0x7f26e406afc0 msgr2=0x7f26e406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- 172.21.15.32:0/2766577197 shutdown_connections 2024-03-16T20:10:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- 172.21.15.32:0/2766577197 wait complete. 2024-03-16T20:10:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 Processor -- start 2024-03-16T20:10:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- start start 2024-03-16T20:10:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e40b1eb0 0x7f26e4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4157680 con 0x7f26e40b1eb0 2024-03-16T20:10:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41e3f70 con 0x7f26e40c7ce0 2024-03-16T20:10:42.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.817+0000 7f26e991e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4159530 con 0x7f26e4158980 2024-03-16T20:10:42.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.821+0000 7f26e27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:42.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.821+0000 7f26e37fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:42.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.821+0000 7f26e27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:51098/0 (socket says 172.21.15.32:51098) 2024-03-16T20:10:42.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.821+0000 7f26e27fc640 1 -- 172.21.15.32:0/3444181512 learned_addr learned my addr 172.21.15.32:0/3444181512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.825+0000 7f26e27fc640 1 -- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 msgr2=0x7f26e4150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.825+0000 7f26e27fc640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.825+0000 7f26e27fc640 1 -- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e40b1eb0 msgr2=0x7f26e4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.825+0000 7f26e27fc640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e40b1eb0 0x7f26e4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.825+0000 7f26e27fc640 1 -- 172.21.15.32:0/3444181512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40094d0 con 0x7f26e40c7ce0 2024-03-16T20:10:42.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.829+0000 7f26e27fc640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b3e0 tx=0x7f26d800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.837+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8011830 con 0x7f26e40c7ce0 2024-03-16T20:10:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.837+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8011990 con 0x7f26e40c7ce0 2024-03-16T20:10:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.837+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8011d60 con 0x7f26e40c7ce0 2024-03-16T20:10:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.837+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e4156de0 con 0x7f26e40c7ce0 2024-03-16T20:10:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.837+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e41573a0 con 0x7f26e40c7ce0 2024-03-16T20:10:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:42.841+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4068000 con 0x7f26e40c7ce0 2024-03-16T20:10:42.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:10:42.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:10:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.157+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f26d801a400 con 0x7f26e40c7ce0 2024-03-16T20:10:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.161+0000 7f26cbfff640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 0x7f26d0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.161+0000 7f26e2ffd640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 0x7f26d0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:43.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.161+0000 7f26e2ffd640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 0x7f26d0069330 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b2000 tx=0x7f26d4004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:43.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.161+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f26d80562f0 con 0x7f26e40c7ce0 2024-03-16T20:10:43.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.185+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f26d8051e90 con 0x7f26e40c7ce0 2024-03-16T20:10:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.489+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e40c60d0 con 0x7f26e40c7ce0 2024-03-16T20:10:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.521+0000 7f26cbfff640 1 -- 172.21.15.32:0/3444181512 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f26d8055550 con 0x7f26e40c7ce0 2024-03-16T20:10:43.511 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:43.511 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 msgr2=0x7f26d0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 0x7f26d0069330 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b2000 tx=0x7f26d4004740 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 msgr2=0x7f26e41f1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b3e0 tx=0x7f26d800b8a0 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 shutdown_connections 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f26d0066e70 0x7f26d0069330 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f26e4158980 0x7f26e4150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40c7ce0 0x7f26e41f1900 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 --2- 172.21.15.32:0/3444181512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e40b1eb0 0x7f26e4158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 >> 172.21.15.32:0/3444181512 conn(0x7f26e406afc0 msgr2=0x7f26e40ca060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 shutdown_connections 2024-03-16T20:10:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:43.525+0000 7f26e991e640 1 -- 172.21.15.32:0/3444181512 wait complete. 2024-03-16T20:10:44.141 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:44.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:45.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 Processor -- start 2024-03-16T20:10:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 -- start start 2024-03-16T20:10:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4fd006c330 0x7f4fd00c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4fd0067120 0x7f4fd00c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd0156cc0 con 0x7f4fd006c710 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd00c14f0 con 0x7f4fd006c330 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.261+0000 7f4fd5b5a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd0156f80 con 0x7f4fd0067120 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.265+0000 7f4fcf7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4fd006c330 0x7f4fd00c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.265+0000 7f4fceffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4fd0067120 0x7f4fd00c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.265+0000 7f4fcffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4fd006c710 0x7f4fd0071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41794/0 (socket says 172.21.15.32:41794) 2024-03-16T20:10:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.265+0000 7f4fcffff640 1 -- 172.21.15.32:0/903604454 learned_addr learned my addr 172.21.15.32:0/903604454 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:45.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.269+0000 7f4fceffd640 1 -- 172.21.15.32:0/903604454 >> v2:172.21.15.32:3300/0 conn(0x7f4fd006c330 msgr2=0x7f4fd00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:45.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.269+0000 7f4fceffd640 1 --2- 172.21.15.32:0/903604454 >> v2:172.21.15.32:3300/0 conn(0x7f4fd006c330 0x7f4fd00c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.269+0000 7f4fceffd640 1 -- 172.21.15.32:0/903604454 >> v1:172.21.15.32:6789/0 conn(0x7f4fd006c710 legacy=0x7f4fd0071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.269+0000 7f4fceffd640 1 -- 172.21.15.32:0/903604454 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd0157260 con 0x7f4fd0067120 2024-03-16T20:10:45.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.273+0000 7f4fceffd640 1 --2- 172.21.15.32:0/903604454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd0067120 0x7f4fd00c2c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0071600 tx=0x7f4fc401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b21290bef97b102 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:45.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fce7fc640 1 -- 172.21.15.32:0/903604454 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40a8970 con 0x7f4fd0067120 2024-03-16T20:10:45.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fce7fc640 1 -- 172.21.15.32:0/903604454 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc40a8ad0 con 0x7f4fd0067120 2024-03-16T20:10:45.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fce7fc640 1 -- 172.21.15.32:0/903604454 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40a8e60 con 0x7f4fd0067120 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/903604454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd0067120 msgr2=0x7f4fd00c2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/903604454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd0067120 0x7f4fd00c2c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0071600 tx=0x7f4fc401ec90 comp rx=0 tx=0).stop 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/903604454 shutdown_connections 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/903604454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd0067120 0x7f4fd00c2c30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/903604454 >> v2:172.21.15.32:3300/0 conn(0x7f4fd006c330 0x7f4fd00c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/903604454 >> 172.21.15.32:0/903604454 conn(0x7f4fd006f3a0 msgr2=0x7f4fd006f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/903604454 shutdown_connections 2024-03-16T20:10:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/903604454 wait complete. 2024-03-16T20:10:45.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 Processor -- start 2024-03-16T20:10:45.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.281+0000 7f4fd5b5a640 1 -- start start 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fd006c330 0x7f4fd01f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd006c710 0x7f4fd0153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd00c8850 con 0x7f4fd0067120 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd014dd50 con 0x7f4fd006c710 2024-03-16T20:10:45.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fd5b5a640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd01ee690 con 0x7f4fd006c330 2024-03-16T20:10:45.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fcf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fceffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fd006c330 0x7f4fd01f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fcf7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42150/0 (socket says 172.21.15.32:42150) 2024-03-16T20:10:45.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2818142425 learned_addr learned my addr 172.21.15.32:0/2818142425 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:45.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.285+0000 7f4fcffff640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd006c710 0x7f4fd0153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.289+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fd006c330 msgr2=0x7f4fd01f15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.289+0000 7f4fcf7fe640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fd006c330 0x7f4fd01f15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.289+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd006c710 msgr2=0x7f4fd0153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.289+0000 7f4fcf7fe640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd006c710 0x7f4fd0153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.289+0000 7f4fcf7fe640 1 -- 172.21.15.32:0/2818142425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4009420 con 0x7f4fd0067120 2024-03-16T20:10:45.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.293+0000 7f4fcf7fe640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e960 tx=0x7f4fc000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:45.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.297+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0009690 con 0x7f4fd0067120 2024-03-16T20:10:45.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.297+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc000ca00 con 0x7f4fd0067120 2024-03-16T20:10:45.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.297+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd01f1150 con 0x7f4fd0067120 2024-03-16T20:10:45.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.297+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000cd70 con 0x7f4fd0067120 2024-03-16T20:10:45.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.297+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd00c8290 con 0x7f4fd0067120 2024-03-16T20:10:45.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.301+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd0068000 con 0x7f4fd0067120 2024-03-16T20:10:45.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.337+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4fc0013470 con 0x7f4fd0067120 2024-03-16T20:10:45.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.337+0000 7f4fb7fff640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 0x7f4fbc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:45.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.337+0000 7f4fceffd640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 0x7f4fbc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:45.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.337+0000 7f4fceffd640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 0x7f4fbc069380 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40b2000 tx=0x7f4fc40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.613+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4fc0056010 con 0x7f4fd0067120 2024-03-16T20:10:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.613+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4fc00522b0 con 0x7f4fd0067120 2024-03-16T20:10:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.885+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fd0154b10 con 0x7f4fd0067120 2024-03-16T20:10:45.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.897+0000 7f4fb7fff640 1 -- 172.21.15.32:0/2818142425 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4fc0021020 con 0x7f4fd0067120 2024-03-16T20:10:45.889 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:45.889 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:45.889 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 msgr2=0x7f4fbc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 0x7f4fbc069380 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40b2000 tx=0x7f4fc40aa070 comp rx=0 tx=0).stop 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 msgr2=0x7f4fd01ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e960 tx=0x7f4fc000ee20 comp rx=0 tx=0).stop 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 shutdown_connections 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4fbc066ec0 0x7f4fbc069380 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fd006c710 0x7f4fd0153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fd006c330 0x7f4fd01f15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 --2- 172.21.15.32:0/2818142425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fd0067120 0x7f4fd01ee150 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 >> 172.21.15.32:0/2818142425 conn(0x7f4fd006f3a0 msgr2=0x7f4fd014e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 shutdown_connections 2024-03-16T20:10:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:45.901+0000 7f4fd5b5a640 1 -- 172.21.15.32:0/2818142425 wait complete. 2024-03-16T20:10:46.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:46.512 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:47.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:47.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 Processor -- start 2024-03-16T20:10:47.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 -- start start 2024-03-16T20:10:47.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f00280c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00280c36b0 0x7f002815a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00280c8ec0 con 0x7f00280c8980 2024-03-16T20:10:47.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0028072f10 con 0x7f00280c36b0 2024-03-16T20:10:47.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.629+0000 7f002e555640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00280c5000 con 0x7f0028151000 2024-03-16T20:10:47.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.633+0000 7f00277fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00280c36b0 0x7f002815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.633+0000 7f0027fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f00280c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.633+0000 7f00277fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00280c36b0 0x7f002815a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:51128/0 (socket says 172.21.15.32:51128) 2024-03-16T20:10:47.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.633+0000 7f00277fe640 1 -- 172.21.15.32:0/4112018041 learned_addr learned my addr 172.21.15.32:0/4112018041 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:47.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.637+0000 7f00277fe640 1 -- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 msgr2=0x7f00280c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:47.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.637+0000 7f00277fe640 1 --2- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f00280c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:47.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.637+0000 7f00277fe640 1 -- 172.21.15.32:0/4112018041 >> v1:172.21.15.59:6789/0 conn(0x7f00280c8980 legacy=0x7f00280c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:47.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.637+0000 7f00277fe640 1 -- 172.21.15.32:0/4112018041 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0028072950 con 0x7f00280c36b0 2024-03-16T20:10:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.641+0000 7f00277fe640 1 --2- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c36b0 0x7f002815a240 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c01c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ea211cfc9fbd69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:47.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.645+0000 7f0026ffd640 1 -- 172.21.15.32:0/4112018041 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c0aa640 con 0x7f00280c36b0 2024-03-16T20:10:47.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f0026ffd640 1 -- 172.21.15.32:0/4112018041 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c0aa7a0 con 0x7f00280c36b0 2024-03-16T20:10:47.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f0026ffd640 1 -- 172.21.15.32:0/4112018041 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c0aab30 con 0x7f00280c36b0 2024-03-16T20:10:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 -- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c36b0 msgr2=0x7f002815a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 --2- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c36b0 0x7f002815a240 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c01c520 comp rx=0 tx=0).stop 2024-03-16T20:10:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 -- 172.21.15.32:0/4112018041 shutdown_connections 2024-03-16T20:10:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 --2- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c36b0 0x7f002815a240 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 --2- 172.21.15.32:0/4112018041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f00280c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 -- 172.21.15.32:0/4112018041 >> 172.21.15.32:0/4112018041 conn(0x7f002806c0d0 msgr2=0x7f002806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 -- 172.21.15.32:0/4112018041 shutdown_connections 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.649+0000 7f002e555640 1 -- 172.21.15.32:0/4112018041 wait complete. 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 Processor -- start 2024-03-16T20:10:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 -- start start 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c8980 0x7f00281f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f002814b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028142a60 con 0x7f00280c36b0 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00281581e0 con 0x7f00280c8980 2024-03-16T20:10:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002e555640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00281f2f20 con 0x7f0028151000 2024-03-16T20:10:47.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f0027fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f0027fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42154/0 (socket says 172.21.15.32:42154) 2024-03-16T20:10:47.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f0027fff640 1 -- 172.21.15.32:0/3022614412 learned_addr learned my addr 172.21.15.32:0/3022614412 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:47.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f00277fe640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c8980 0x7f00281f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.653+0000 7f002caf2640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f002814b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.657+0000 7f0027fff640 1 -- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 msgr2=0x7f002814b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.657+0000 7f0027fff640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f002814b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.657+0000 7f0027fff640 1 -- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c8980 msgr2=0x7f00281f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.657+0000 7f0027fff640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c8980 0x7f00281f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.657+0000 7f0027fff640 1 -- 172.21.15.32:0/3022614412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0094d0 con 0x7f00280c36b0 2024-03-16T20:10:47.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.661+0000 7f0027fff640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f001800e870 tx=0x7f001800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018009530 con 0x7f00280c36b0 2024-03-16T20:10:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001800c940 con 0x7f00280c36b0 2024-03-16T20:10:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001800ccd0 con 0x7f00280c36b0 2024-03-16T20:10:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00281f68b0 con 0x7f00280c36b0 2024-03-16T20:10:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00281f6e70 con 0x7f00280c36b0 2024-03-16T20:10:47.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.665+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0028068000 con 0x7f00280c36b0 2024-03-16T20:10:47.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.689+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0018013470 con 0x7f00280c36b0 2024-03-16T20:10:47.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.689+0000 7f0024ff9640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 0x7f0014069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:47.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.689+0000 7f00277fe640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 0x7f0014069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:47.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.693+0000 7f00277fe640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 0x7f0014069300 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f001c0ae000 tx=0x7f001c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.957+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f0018055ec0 con 0x7f00280c36b0 2024-03-16T20:10:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:47.957+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f0018052160 con 0x7f00280c36b0 2024-03-16T20:10:48.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.249+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0028073410 con 0x7f00280c36b0 2024-03-16T20:10:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.273+0000 7f0024ff9640 1 -- 172.21.15.32:0/3022614412 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f0018017090 con 0x7f00280c36b0 2024-03-16T20:10:48.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:48.266 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 msgr2=0x7f0014069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 0x7f0014069300 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f001c0ae000 tx=0x7f001c0b5040 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 msgr2=0x7f00281421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f001800e870 tx=0x7f001800ed30 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 shutdown_connections 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f0014066e40 0x7f0014069300 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0028151000 0x7f002814b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00280c8980 0x7f00281f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 --2- 172.21.15.32:0/3022614412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00280c36b0 0x7f00281421a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 >> 172.21.15.32:0/3022614412 conn(0x7f002806c0d0 msgr2=0x7f00280c9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 shutdown_connections 2024-03-16T20:10:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:48.277+0000 7f002e555640 1 -- 172.21.15.32:0/3022614412 wait complete. 2024-03-16T20:10:48.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:48.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:49.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:49.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 Processor -- start 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 -- start start 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6e4c0c6040 0x7f6e4c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c0c6420 0x7f6e4c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c070760 con 0x7f6e4c06ba30 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0731c0 con 0x7f6e4c0c6040 2024-03-16T20:10:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.005+0000 7f6e53a37640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0734a0 con 0x7f6e4c0c6420 2024-03-16T20:10:49.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.009+0000 7f6e517d3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6e4c0c6040 0x7f6e4c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.009+0000 7f6e50fd2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c0c6420 0x7f6e4c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.009+0000 7f6e517d3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6e4c0c6040 0x7f6e4c0c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:41236/0 (socket says 172.21.15.32:41236) 2024-03-16T20:10:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.009+0000 7f6e517d3640 1 -- 172.21.15.32:0/121612617 learned_addr learned my addr 172.21.15.32:0/121612617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.013+0000 7f6e517d3640 1 -- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c0c6420 msgr2=0x7f6e4c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.013+0000 7f6e517d3640 1 --2- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c0c6420 0x7f6e4c157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.013+0000 7f6e517d3640 1 -- 172.21.15.32:0/121612617 >> v1:172.21.15.59:6789/0 conn(0x7f6e4c06ba30 legacy=0x7f6e4c1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.013+0000 7f6e517d3640 1 -- 172.21.15.32:0/121612617 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c06fe30 con 0x7f6e4c0c6040 2024-03-16T20:10:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.017+0000 7f6e517d3640 1 --2- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c0c8800 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009870 tx=0x7f6e4801c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7f28d5f6a7819b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:50.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e43fff640 1 -- 172.21.15.32:0/121612617 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480ac620 con 0x7f6e4c0c6040 2024-03-16T20:10:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e43fff640 1 -- 172.21.15.32:0/121612617 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e480ac780 con 0x7f6e4c0c6040 2024-03-16T20:10:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e43fff640 1 -- 172.21.15.32:0/121612617 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480acb10 con 0x7f6e4c0c6040 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 -- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6040 msgr2=0x7f6e4c0c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 --2- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c0c8800 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009870 tx=0x7f6e4801c4f0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 -- 172.21.15.32:0/121612617 shutdown_connections 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 --2- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c0c6420 0x7f6e4c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 --2- 172.21.15.32:0/121612617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c0c8800 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 -- 172.21.15.32:0/121612617 >> 172.21.15.32:0/121612617 conn(0x7f6e4c06c0d0 msgr2=0x7f6e4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 -- 172.21.15.32:0/121612617 shutdown_connections 2024-03-16T20:10:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.025+0000 7f6e53a37640 1 -- 172.21.15.32:0/121612617 wait complete. 2024-03-16T20:10:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 Processor -- start 2024-03-16T20:10:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 -- start start 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c06ba30 0x7f6e4c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6420 0x7f6e4c144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c147dc0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1f4ee0 con 0x7f6e4c0c6420 2024-03-16T20:10:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e53a37640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1f49e0 con 0x7f6e4c06ba30 2024-03-16T20:10:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e50fd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e51fd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6420 0x7f6e4c144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.029+0000 7f6e517d3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c06ba30 0x7f6e4c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.033+0000 7f6e50fd2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51828/0 (socket says 172.21.15.32:51828) 2024-03-16T20:10:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.033+0000 7f6e50fd2640 1 -- 172.21.15.32:0/4070900693 learned_addr learned my addr 172.21.15.32:0/4070900693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.033+0000 7f6e50fd2640 1 -- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c06ba30 msgr2=0x7f6e4c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.033+0000 7f6e50fd2640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c06ba30 0x7f6e4c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.037+0000 7f6e50fd2640 1 -- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6420 msgr2=0x7f6e4c144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.037+0000 7f6e50fd2640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6420 0x7f6e4c144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.037+0000 7f6e50fd2640 1 -- 172.21.15.32:0/4070900693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480094d0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.037+0000 7f6e50fd2640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00e870 tx=0x7f6e3c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:50.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.041+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c009510 con 0x7f6e4c0c6040 2024-03-16T20:10:50.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.041+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c15a4a0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.041+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c157e10 con 0x7f6e4c0c6040 2024-03-16T20:10:50.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.041+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c06fa60 con 0x7f6e4c0c6040 2024-03-16T20:10:50.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.045+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c00eec0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.045+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c005ba0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.389+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6e3c00c940 con 0x7f6e4c0c6040 2024-03-16T20:10:50.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.389+0000 7f6e41ffb640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 0x7f6e30069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:50.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.389+0000 7f6e517d3640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 0x7f6e30069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:50.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.389+0000 7f6e517d3640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 0x7f6e30069330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6e480ae000 tx=0x7f6e480a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:50.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.417+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6e3c055ce0 con 0x7f6e4c0c6040 2024-03-16T20:10:50.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.417+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6e3c051f80 con 0x7f6e4c0c6040 2024-03-16T20:10:50.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.637+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e4c1f4c00 con 0x7f6e4c0c6040 2024-03-16T20:10:50.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.653+0000 7f6e41ffb640 1 -- 172.21.15.32:0/4070900693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6e4c1f4c00 con 0x7f6e4c0c6040 2024-03-16T20:10:50.645 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:50.645 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:50.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 msgr2=0x7f6e30069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 0x7f6e30069330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6e480ae000 tx=0x7f6e480a7000 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 msgr2=0x7f6e4c158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00e870 tx=0x7f6e3c00ec70 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 shutdown_connections 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6e30066e70 0x7f6e30069330 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6e4c0c6420 0x7f6e4c144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c0c6040 0x7f6e4c158e40 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 --2- 172.21.15.32:0/4070900693 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6e4c06ba30 0x7f6e4c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 >> 172.21.15.32:0/4070900693 conn(0x7f6e4c06c0d0 msgr2=0x7f6e4c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 shutdown_connections 2024-03-16T20:10:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:50.657+0000 7f6e53a37640 1 -- 172.21.15.32:0/4070900693 wait complete. 2024-03-16T20:10:51.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:51.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:52.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:52.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 Processor -- start 2024-03-16T20:10:52.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 -- start start 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3b5c0b1eb0 0x7f3b5c072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b5c158980 0x7f3b5c072f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0c68e0 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c067340 con 0x7f3b5c158980 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b61ac8640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c06c1f0 con 0x7f3b5c0b1eb0 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b5bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b5c158980 0x7f3b5c072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b5bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b5c158980 0x7f3b5c072f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51836/0 (socket says 172.21.15.32:51836) 2024-03-16T20:10:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.393+0000 7f3b5bfff640 1 -- 172.21.15.32:0/582445531 learned_addr learned my addr 172.21.15.32:0/582445531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.401+0000 7f3b5b7fe640 1 --2- 172.21.15.32:0/582445531 >> v2:172.21.15.59:3300/0 conn(0x7f3b5c0b1eb0 0x7f3b5c072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.401+0000 7f3b5bfff640 1 -- 172.21.15.32:0/582445531 >> v2:172.21.15.59:3300/0 conn(0x7f3b5c0b1eb0 msgr2=0x7f3b5c072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.401+0000 7f3b5bfff640 1 --2- 172.21.15.32:0/582445531 >> v2:172.21.15.59:3300/0 conn(0x7f3b5c0b1eb0 0x7f3b5c072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.401+0000 7f3b5bfff640 1 -- 172.21.15.32:0/582445531 >> v1:172.21.15.59:6789/0 conn(0x7f3b5c0c7ce0 legacy=0x7f3b5c158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.401+0000 7f3b5bfff640 1 -- 172.21.15.32:0/582445531 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c06c4d0 con 0x7f3b5c158980 2024-03-16T20:10:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.405+0000 7f3b5bfff640 1 --2- 172.21.15.32:0/582445531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c158980 0x7f3b5c072f50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c074a60 tx=0x7f3b5401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b165f2b872af55fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:52.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.409+0000 7f3b5affd640 1 -- 172.21.15.32:0/582445531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b540af400 con 0x7f3b5c158980 2024-03-16T20:10:52.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.409+0000 7f3b5affd640 1 -- 172.21.15.32:0/582445531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b540ab050 con 0x7f3b5c158980 2024-03-16T20:10:52.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.409+0000 7f3b5affd640 1 -- 172.21.15.32:0/582445531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b540af780 con 0x7f3b5c158980 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- 172.21.15.32:0/582445531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c158980 msgr2=0x7f3b5c072f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/582445531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c158980 0x7f3b5c072f50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c074a60 tx=0x7f3b5401f1a0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- 172.21.15.32:0/582445531 shutdown_connections 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/582445531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c158980 0x7f3b5c072f50 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/582445531 >> v2:172.21.15.59:3300/0 conn(0x7f3b5c0b1eb0 0x7f3b5c072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- 172.21.15.32:0/582445531 >> 172.21.15.32:0/582445531 conn(0x7f3b5c06afc0 msgr2=0x7f3b5c06b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:52.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- 172.21.15.32:0/582445531 shutdown_connections 2024-03-16T20:10:52.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- 172.21.15.32:0/582445531 wait complete. 2024-03-16T20:10:52.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 Processor -- start 2024-03-16T20:10:52.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- start start 2024-03-16T20:10:52.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b5c0b1eb0 0x7f3b5c13c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b5c158980 0x7f3b5c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c146770 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c145ff0 con 0x7f3b5c0b1eb0 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.413+0000 7f3b61ac8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c155e90 con 0x7f3b5c158980 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.417+0000 7f3b53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.417+0000 7f3b53fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51850/0 (socket says 172.21.15.32:51850) 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.417+0000 7f3b53fff640 1 -- 172.21.15.32:0/669723524 learned_addr learned my addr 172.21.15.32:0/669723524 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.417+0000 7f3b5bfff640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b5c158980 0x7f3b5c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.417+0000 7f3b5b7fe640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b5c0b1eb0 0x7f3b5c13c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 -- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b5c158980 msgr2=0x7f3b5c156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:52.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b5c158980 0x7f3b5c156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 -- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b5c0b1eb0 msgr2=0x7f3b5c13c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:52.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b5c0b1eb0 0x7f3b5c13c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:52.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 -- 172.21.15.32:0/669723524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54009b30 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.421+0000 7f3b53fff640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c410 tx=0x7f3b4400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:52.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.425+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003600 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.425+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b44003780 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.425+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003b30 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.425+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c13c760 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.425+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c145470 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.429+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c068000 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.457+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3b44003c90 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.457+0000 7f3b58ff9640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 0x7f3b3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:52.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.457+0000 7f3b5b7fe640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 0x7f3b3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:52.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.461+0000 7f3b5b7fe640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 0x7f3b3c069300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006fd0 tx=0x7f3b4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.733+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3b440010f0 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.733+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3b44051070 con 0x7f3b5c0c7ce0 2024-03-16T20:10:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:52.961+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b5c0c60d0 con 0x7f3b5c0c7ce0 2024-03-16T20:10:53.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.037+0000 7f3b58ff9640 1 -- 172.21.15.32:0/669723524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f3b44054730 con 0x7f3b5c0c7ce0 2024-03-16T20:10:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:53.028 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:53.028 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 msgr2=0x7f3b3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 0x7f3b3c069300 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006fd0 tx=0x7f3b4c008040 comp rx=0 tx=0).stop 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 msgr2=0x7f3b5c147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c410 tx=0x7f3b4400c8d0 comp rx=0 tx=0).stop 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 shutdown_connections 2024-03-16T20:10:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3b3c066e40 0x7f3b3c069300 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3b5c158980 0x7f3b5c156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b5c0c7ce0 0x7f3b5c147870 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 --2- 172.21.15.32:0/669723524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b5c0b1eb0 0x7f3b5c13c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 >> 172.21.15.32:0/669723524 conn(0x7f3b5c06afc0 msgr2=0x7f3b5c0ca970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 shutdown_connections 2024-03-16T20:10:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:53.041+0000 7f3b61ac8640 1 -- 172.21.15.32:0/669723524 wait complete. 2024-03-16T20:10:53.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:53.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:54.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:54.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 Processor -- start 2024-03-16T20:10:54.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 -- start start 2024-03-16T20:10:54.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb180c81e0 0x7ffb180c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffb180c3570 0x7ffb180c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1806c460 con 0x7ffb181540c0 2024-03-16T20:10:54.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb180c3f40 con 0x7ffb180c81e0 2024-03-16T20:10:54.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.769+0000 7ffb1dd7c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1815a240 con 0x7ffb180c3570 2024-03-16T20:10:54.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.773+0000 7ffb17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffb180c3570 0x7ffb180c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:54.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.773+0000 7ffb17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffb180c3570 0x7ffb180c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:41284/0 (socket says 172.21.15.32:41284) 2024-03-16T20:10:54.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.773+0000 7ffb17fff640 1 -- 172.21.15.32:0/2346992546 learned_addr learned my addr 172.21.15.32:0/2346992546 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:54.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.777+0000 7ffb17fff640 1 -- 172.21.15.32:0/2346992546 >> v2:172.21.15.32:3300/0 conn(0x7ffb180c81e0 msgr2=0x7ffb180c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:54.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.777+0000 7ffb17fff640 1 --2- 172.21.15.32:0/2346992546 >> v2:172.21.15.32:3300/0 conn(0x7ffb180c81e0 0x7ffb180c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:54.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.777+0000 7ffb17fff640 1 -- 172.21.15.32:0/2346992546 >> v1:172.21.15.59:6789/0 conn(0x7ffb181540c0 legacy=0x7ffb1815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:54.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.777+0000 7ffb17fff640 1 -- 172.21.15.32:0/2346992546 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1806cf20 con 0x7ffb180c3570 2024-03-16T20:10:54.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.781+0000 7ffb17fff640 1 --2- 172.21.15.32:0/2346992546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb180c3570 0x7ffb180c3a00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffb1815a1b0 tx=0x7ffb1001c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc15df7d0fb3a77e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:54.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb167fc640 1 -- 172.21.15.32:0/2346992546 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100a5840 con 0x7ffb180c3570 2024-03-16T20:10:54.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb167fc640 1 -- 172.21.15.32:0/2346992546 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb100a59a0 con 0x7ffb180c3570 2024-03-16T20:10:54.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb167fc640 1 -- 172.21.15.32:0/2346992546 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100a5d30 con 0x7ffb180c3570 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2346992546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb180c3570 msgr2=0x7ffb180c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2346992546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb180c3570 0x7ffb180c3a00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffb1815a1b0 tx=0x7ffb1001c980 comp rx=0 tx=0).stop 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2346992546 shutdown_connections 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2346992546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb180c3570 0x7ffb180c3a00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2346992546 >> v2:172.21.15.32:3300/0 conn(0x7ffb180c81e0 0x7ffb180c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2346992546 >> 172.21.15.32:0/2346992546 conn(0x7ffb18070f90 msgr2=0x7ffb180713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2346992546 shutdown_connections 2024-03-16T20:10:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.789+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2346992546 wait complete. 2024-03-16T20:10:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 Processor -- start 2024-03-16T20:10:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 -- start start 2024-03-16T20:10:54.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffb180c81e0 0x7ffb1815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb181540c0 0x7ffb1815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18152ff0 con 0x7ffb180c3570 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18144cd0 con 0x7ffb181540c0 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb1dd7c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18151830 con 0x7ffb180c81e0 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51880/0 (socket says 172.21.15.32:51880) 2024-03-16T20:10:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb177fe640 1 -- 172.21.15.32:0/2906872785 learned_addr learned my addr 172.21.15.32:0/2906872785 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb17fff640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb181540c0 0x7ffb1815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.793+0000 7ffb16ffd640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffb180c81e0 0x7ffb1815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.797+0000 7ffb177fe640 1 -- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffb180c81e0 msgr2=0x7ffb1815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.797+0000 7ffb177fe640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffb180c81e0 0x7ffb1815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.797+0000 7ffb177fe640 1 -- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb181540c0 msgr2=0x7ffb1815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.797+0000 7ffb177fe640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb181540c0 0x7ffb1815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.797+0000 7ffb177fe640 1 -- 172.21.15.32:0/2906872785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009b30 con 0x7ffb180c3570 2024-03-16T20:10:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.801+0000 7ffb177fe640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c490 tx=0x7ffb08001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08003600 con 0x7ffb180c3570 2024-03-16T20:10:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18151a60 con 0x7ffb180c3570 2024-03-16T20:10:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb18152000 con 0x7ffb180c3570 2024-03-16T20:10:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18070390 con 0x7ffb180c3570 2024-03-16T20:10:54.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb08001130 con 0x7ffb180c3570 2024-03-16T20:10:54.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.805+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08007230 con 0x7ffb180c3570 2024-03-16T20:10:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.829+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ffb08007390 con 0x7ffb180c3570 2024-03-16T20:10:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.829+0000 7ffafffff640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 0x7ffb04069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:54.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.829+0000 7ffb16ffd640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 0x7ffb04069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:54.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:54.833+0000 7ffb16ffd640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 0x7ffb04069300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ffb1806dfa0 tx=0x7ffb0c009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:55.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.113+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ffb08007640 con 0x7ffb180c3570 2024-03-16T20:10:55.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.113+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ffb08051380 con 0x7ffb180c3570 2024-03-16T20:10:55.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.393+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb1815b330 con 0x7ffb180c3570 2024-03-16T20:10:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.453+0000 7ffafffff640 1 -- 172.21.15.32:0/2906872785 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7ffb08017020 con 0x7ffb180c3570 2024-03-16T20:10:55.445 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:55.445 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 msgr2=0x7ffb04069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 0x7ffb04069300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ffb1806dfa0 tx=0x7ffb0c009110 comp rx=0 tx=0).stop 2024-03-16T20:10:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 msgr2=0x7ffb181468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c490 tx=0x7ffb08001e80 comp rx=0 tx=0).stop 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 shutdown_connections 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ffb04066e40 0x7ffb04069300 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb181540c0 0x7ffb1815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffb180c81e0 0x7ffb1815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 --2- 172.21.15.32:0/2906872785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb180c3570 0x7ffb181468a0 secure :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c490 tx=0x7ffb08001e80 comp rx=0 tx=0).stop 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 >> 172.21.15.32:0/2906872785 conn(0x7ffb18070f90 msgr2=0x7ffb1806efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 shutdown_connections 2024-03-16T20:10:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:55.457+0000 7ffb1dd7c640 1 -- 172.21.15.32:0/2906872785 wait complete. 2024-03-16T20:10:56.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:56.070 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:57.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.189+0000 7fb042dbe640 1 Processor -- start 2024-03-16T20:10:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.189+0000 7fb042dbe640 1 -- start start 2024-03-16T20:10:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb042dbe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb03c153fa0 0x7fb03c070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb042dbe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 0x7fb03c072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb042dbe640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0c8590 con 0x7fb03c0c7470 2024-03-16T20:10:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb042dbe640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0c3f00 con 0x7fb03c153fa0 2024-03-16T20:10:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb042dbe640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c070650 con 0x7fb03c0724a0 2024-03-16T20:10:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb04135b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 0x7fb03c072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb03bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb03c0c7470 0x7fb03c15d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:48270/0 (socket says 172.21.15.32:48270) 2024-03-16T20:10:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb03bfff640 1 -- 172.21.15.32:0/1960758124 learned_addr learned my addr 172.21.15.32:0/1960758124 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.193+0000 7fb040b5a640 1 --2- 172.21.15.32:0/1960758124 >> v2:172.21.15.59:3300/0 conn(0x7fb03c153fa0 0x7fb03c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.201+0000 7fb04135b640 1 -- 172.21.15.32:0/1960758124 >> v2:172.21.15.59:3300/0 conn(0x7fb03c153fa0 msgr2=0x7fb03c070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:10:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.201+0000 7fb04135b640 1 --2- 172.21.15.32:0/1960758124 >> v2:172.21.15.59:3300/0 conn(0x7fb03c153fa0 0x7fb03c070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.201+0000 7fb04135b640 1 -- 172.21.15.32:0/1960758124 >> v1:172.21.15.59:6789/0 conn(0x7fb03c0c7470 legacy=0x7fb03c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.201+0000 7fb04135b640 1 -- 172.21.15.32:0/1960758124 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c073590 con 0x7fb03c0724a0 2024-03-16T20:10:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.205+0000 7fb04135b640 1 --2- 172.21.15.32:0/1960758124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 0x7fb03c072930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb034009d90 tx=0x7fb03401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2d8bdf1fc09945 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb03b7fe640 1 -- 172.21.15.32:0/1960758124 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0340af600 con 0x7fb03c0724a0 2024-03-16T20:10:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb03b7fe640 1 -- 172.21.15.32:0/1960758124 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0340ab050 con 0x7fb03c0724a0 2024-03-16T20:10:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb03b7fe640 1 -- 172.21.15.32:0/1960758124 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0340af980 con 0x7fb03c0724a0 2024-03-16T20:10:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 -- 172.21.15.32:0/1960758124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 msgr2=0x7fb03c072930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 --2- 172.21.15.32:0/1960758124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 0x7fb03c072930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb034009d90 tx=0x7fb03401f150 comp rx=0 tx=0).stop 2024-03-16T20:10:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 -- 172.21.15.32:0/1960758124 shutdown_connections 2024-03-16T20:10:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 --2- 172.21.15.32:0/1960758124 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0724a0 0x7fb03c072930 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 --2- 172.21.15.32:0/1960758124 >> v2:172.21.15.59:3300/0 conn(0x7fb03c153fa0 0x7fb03c070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 -- 172.21.15.32:0/1960758124 >> 172.21.15.32:0/1960758124 conn(0x7fb03c06c0d0 msgr2=0x7fb03c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 -- 172.21.15.32:0/1960758124 shutdown_connections 2024-03-16T20:10:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.217+0000 7fb042dbe640 1 -- 172.21.15.32:0/1960758124 wait complete. 2024-03-16T20:10:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 Processor -- start 2024-03-16T20:10:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 -- start start 2024-03-16T20:10:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb03c0724a0 0x7fb03c15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0c7470 0x7fb03c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c143e80 con 0x7fb03c153fa0 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c15ce40 con 0x7fb03c0724a0 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb042dbe640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c15dfb0 con 0x7fb03c0c7470 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb04135b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb04135b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51884/0 (socket says 172.21.15.32:51884) 2024-03-16T20:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb040b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb03c0724a0 0x7fb03c15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb04135b640 1 -- 172.21.15.32:0/3693436452 learned_addr learned my addr 172.21.15.32:0/3693436452 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:57.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.221+0000 7fb03bfff640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0c7470 0x7fb03c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.225+0000 7fb04135b640 1 -- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0c7470 msgr2=0x7fb03c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.225+0000 7fb04135b640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0c7470 0x7fb03c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.225+0000 7fb04135b640 1 -- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb03c0724a0 msgr2=0x7fb03c15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.225+0000 7fb04135b640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb03c0724a0 0x7fb03c15da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.225+0000 7fb04135b640 1 -- 172.21.15.32:0/3693436452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009a70 con 0x7fb03c153fa0 2024-03-16T20:10:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.229+0000 7fb04135b640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb03401cd80 tx=0x7fb03401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:57.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.233+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0340afca0 con 0x7fb03c153fa0 2024-03-16T20:10:57.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.233+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0340afe00 con 0x7fb03c153fa0 2024-03-16T20:10:57.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.233+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c1efdb0 con 0x7fb03c153fa0 2024-03-16T20:10:57.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.233+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0340ba660 con 0x7fb03c153fa0 2024-03-16T20:10:57.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.233+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c1f0370 con 0x7fb03c153fa0 2024-03-16T20:10:57.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.237+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c068000 con 0x7fb03c153fa0 2024-03-16T20:10:57.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.257+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb0340ba7c0 con 0x7fb03c153fa0 2024-03-16T20:10:57.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.257+0000 7fb0397fa640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 0x7fb020069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:57.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.257+0000 7fb040b5a640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 0x7fb020069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:57.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.261+0000 7fb040b5a640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 0x7fb020069380 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ac0 tx=0x7fb030009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.549+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb0340f4090 con 0x7fb03c153fa0 2024-03-16T20:10:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.549+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fb0340f0330 con 0x7fb03c153fa0 2024-03-16T20:10:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.793+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb03c0c6f90 con 0x7fb03c153fa0 2024-03-16T20:10:57.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.845+0000 7fb0397fa640 1 -- 172.21.15.32:0/3693436452 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fb0340be090 con 0x7fb03c153fa0 2024-03-16T20:10:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:10:57.838 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:10:57.838 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 msgr2=0x7fb020069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 0x7fb020069380 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ac0 tx=0x7fb030009190 comp rx=0 tx=0).stop 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 msgr2=0x7fb03c0cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb03401cd80 tx=0x7fb03401bef0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 shutdown_connections 2024-03-16T20:10:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fb020066ec0 0x7fb020069380 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb03c153fa0 0x7fb03c0cca40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb03c0c7470 0x7fb03c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 --2- 172.21.15.32:0/3693436452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb03c0724a0 0x7fb03c15da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 >> 172.21.15.32:0/3693436452 conn(0x7fb03c06c0d0 msgr2=0x7fb03c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 shutdown_connections 2024-03-16T20:10:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:57.849+0000 7fb042dbe640 1 -- 172.21.15.32:0/3693436452 wait complete. 2024-03-16T20:10:58.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:10:58.468 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:10:59.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:10:59.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 Processor -- start 2024-03-16T20:10:59.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 -- start start 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44c00c7830 0x7f44c0158340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c0157eb0 con 0x7f44c00cdb80 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00c4c80 con 0x7f44c00cdf60 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44c5bb1640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c006fa80 con 0x7f44c00c7830 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44bffff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44c00c7830 0x7f44c0158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44bf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f44c00cdb80 0x7f44c006f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36780/0 (socket says 172.21.15.32:36780) 2024-03-16T20:10:59.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.589+0000 7f44bf7fe640 1 -- 172.21.15.32:0/2641619504 learned_addr learned my addr 172.21.15.32:0/2641619504 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:10:59.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.593+0000 7f44bffff640 1 -- 172.21.15.32:0/2641619504 >> v1:172.21.15.59:6789/0 conn(0x7f44c00cdf60 legacy=0x7f44c0151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:59.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.593+0000 7f44bffff640 1 -- 172.21.15.32:0/2641619504 >> v1:172.21.15.32:6789/0 conn(0x7f44c00cdb80 legacy=0x7f44c006f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:59.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.593+0000 7f44bffff640 1 -- 172.21.15.32:0/2641619504 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c00c4f40 con 0x7f44c00c7830 2024-03-16T20:10:59.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.601+0000 7f44bffff640 1 --2- 172.21.15.32:0/2641619504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00c7830 0x7f44c0158340 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009d90 tx=0x7f44b801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c7998e0747ca00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:59.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44be7fc640 1 -- 172.21.15.32:0/2641619504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80a5910 con 0x7f44c00c7830 2024-03-16T20:10:59.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44be7fc640 1 -- 172.21.15.32:0/2641619504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b80a5a70 con 0x7f44c00c7830 2024-03-16T20:10:59.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44be7fc640 1 -- 172.21.15.32:0/2641619504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80a5e00 con 0x7f44c00c7830 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2641619504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00c7830 msgr2=0x7f44c0158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 --2- 172.21.15.32:0/2641619504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00c7830 0x7f44c0158340 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009d90 tx=0x7f44b801c940 comp rx=0 tx=0).stop 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2641619504 shutdown_connections 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 --2- 172.21.15.32:0/2641619504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00c7830 0x7f44c0158340 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2641619504 >> 172.21.15.32:0/2641619504 conn(0x7f44c0066ea0 msgr2=0x7f44c00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:10:59.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2641619504 shutdown_connections 2024-03-16T20:10:59.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.605+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2641619504 wait complete. 2024-03-16T20:10:59.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 Processor -- start 2024-03-16T20:10:59.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 -- start start 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44c00c7830 0x7f44c0140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44c00cdf60 0x7f44c00cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006c200 con 0x7f44c00cdb80 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006def0 con 0x7f44c00cdf60 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44c5bb1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c013bad0 con 0x7f44c00c7830 2024-03-16T20:10:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44beffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44beffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56914/0 (socket says 172.21.15.32:56914) 2024-03-16T20:10:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44beffd640 1 -- 172.21.15.32:0/2367652335 learned_addr learned my addr 172.21.15.32:0/2367652335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:10:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.609+0000 7f44bffff640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44c00cdf60 0x7f44c00cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44bf7fe640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44c00c7830 0x7f44c0140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44beffd640 1 -- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44c00c7830 msgr2=0x7f44c0140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44beffd640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44c00c7830 0x7f44c0140290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44beffd640 1 -- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44c00cdf60 msgr2=0x7f44c00cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:10:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44beffd640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44c00cdf60 0x7f44c00cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:10:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.613+0000 7f44beffd640 1 -- 172.21.15.32:0/2367652335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8009a70 con 0x7f44c00cdb80 2024-03-16T20:10:59.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.617+0000 7f44beffd640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c500 tx=0x7f44b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4005220 con 0x7f44c00cdb80 2024-03-16T20:10:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b4005380 con 0x7f44c00cdb80 2024-03-16T20:10:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2367652335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c013bdb0 con 0x7f44c00cdb80 2024-03-16T20:10:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4007270 con 0x7f44c00cdb80 2024-03-16T20:10:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44c5bb1640 1 -- 172.21.15.32:0/2367652335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01407d0 con 0x7f44c00cdb80 2024-03-16T20:10:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.621+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44980052a0 con 0x7f44c00cdb80 2024-03-16T20:10:59.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.897+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f44b40073d0 con 0x7f44c00cdb80 2024-03-16T20:10:59.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.897+0000 7f44a7fff640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 0x7f44ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:10:59.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.897+0000 7f44bf7fe640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 0x7f44ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:10:59.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.901+0000 7f44bf7fe640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 0x7f44ac069300 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f70 tx=0x7f44b0007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:10:59.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.925+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f44b4007680 con 0x7f44c00cdb80 2024-03-16T20:10:59.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:10:59.925+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f44b4051f00 con 0x7f44c00cdb80 2024-03-16T20:11:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.181+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4498005900 con 0x7f44c00cdb80 2024-03-16T20:11:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.205+0000 7f44a7fff640 1 -- 172.21.15.32:0/2367652335 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f4498005900 con 0x7f44c00cdb80 2024-03-16T20:11:00.198 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:00.199 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:00.199 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 msgr2=0x7f44ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 0x7f44ac069300 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f70 tx=0x7f44b0007e90 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 msgr2=0x7f44c006c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c500 tx=0x7f44b400c9c0 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 shutdown_connections 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f44ac066e40 0x7f44ac069300 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44c00cdf60 0x7f44c00cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c00cdb80 0x7f44c006c650 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 --2- 172.21.15.32:0/2367652335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44c00c7830 0x7f44c0140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:00.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 >> 172.21.15.32:0/2367652335 conn(0x7f44c0066ea0 msgr2=0x7f44c00c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:00.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 shutdown_connections 2024-03-16T20:11:00.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:00.209+0000 7f44a5ffb640 1 -- 172.21.15.32:0/2367652335 wait complete. 2024-03-16T20:11:00.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:00.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:01.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.937+0000 7f020c49b640 1 Processor -- start 2024-03-16T20:11:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.937+0000 7f020c49b640 1 -- start start 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020c49b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02040c36b0 0x7f02040c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020c49b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02040c8980 0x7f020415a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020c49b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02040c8ec0 con 0x7f0204151000 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020c49b640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0204072f10 con 0x7f02040c36b0 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020c49b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02040c5000 con 0x7f02040c8980 2024-03-16T20:11:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f0209a36640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02040c36b0 0x7f02040c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f0209a36640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f02040c36b0 0x7f02040c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56922/0 (socket says 172.21.15.32:56922) 2024-03-16T20:11:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f0209a36640 1 -- 172.21.15.32:0/3426511437 learned_addr learned my addr 172.21.15.32:0/3426511437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.941+0000 7f020aa38640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02040c8980 0x7f020415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.945+0000 7f0209a36640 1 -- 172.21.15.32:0/3426511437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02040c8980 msgr2=0x7f020415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.945+0000 7f0209a36640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02040c8980 0x7f020415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.945+0000 7f0209a36640 1 -- 172.21.15.32:0/3426511437 >> v1:172.21.15.59:6789/0 conn(0x7f0204151000 legacy=0x7f02040c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.945+0000 7f0209a36640 1 -- 172.21.15.32:0/3426511437 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0204072950 con 0x7f02040c36b0 2024-03-16T20:11:01.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.945+0000 7f0209a36640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c36b0 0x7f02040c8420 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f401c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6194cb035c45d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:01.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f0209235640 1 -- 172.21.15.32:0/3426511437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f40a9640 con 0x7f02040c36b0 2024-03-16T20:11:01.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f0209235640 1 -- 172.21.15.32:0/3426511437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f40a97a0 con 0x7f02040c36b0 2024-03-16T20:11:01.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f0209235640 1 -- 172.21.15.32:0/3426511437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f40a9b70 con 0x7f02040c36b0 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 -- 172.21.15.32:0/3426511437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c36b0 msgr2=0x7f02040c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c36b0 0x7f02040c8420 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f401c580 comp rx=0 tx=0).stop 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 -- 172.21.15.32:0/3426511437 shutdown_connections 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f02040c8980 0x7f020415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 --2- 172.21.15.32:0/3426511437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c36b0 0x7f02040c8420 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 -- 172.21.15.32:0/3426511437 >> 172.21.15.32:0/3426511437 conn(0x7f020406c0d0 msgr2=0x7f020406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:01.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 -- 172.21.15.32:0/3426511437 shutdown_connections 2024-03-16T20:11:01.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.953+0000 7f020c49b640 1 -- 172.21.15.32:0/3426511437 wait complete. 2024-03-16T20:11:01.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 Processor -- start 2024-03-16T20:11:01.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 -- start start 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02040c36b0 0x7f0204160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0204151000 0x7f02041f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020415a610 con 0x7f02040c8980 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020414b7c0 con 0x7f02040c36b0 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020c49b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02041611c0 con 0x7f0204151000 2024-03-16T20:11:01.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f0209a36640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f0209a36640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56934/0 (socket says 172.21.15.32:56934) 2024-03-16T20:11:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f0209a36640 1 -- 172.21.15.32:0/2201753710 learned_addr learned my addr 172.21.15.32:0/2201753710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020aa38640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0204151000 0x7f02041f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.957+0000 7f020a237640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02040c36b0 0x7f0204160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.961+0000 7f0209a36640 1 -- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0204151000 msgr2=0x7f02041f6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.961+0000 7f0209a36640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0204151000 0x7f02041f6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.961+0000 7f0209a36640 1 -- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02040c36b0 msgr2=0x7f0204160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.961+0000 7f0209a36640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02040c36b0 0x7f0204160660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.961+0000 7f0209a36640 1 -- 172.21.15.32:0/2201753710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f40094d0 con 0x7f02040c8980 2024-03-16T20:11:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.965+0000 7f0209a36640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f01f40ab040 tx=0x7f01f401adc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f40bb420 con 0x7f02040c8980 2024-03-16T20:11:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f40bb5a0 con 0x7f02040c8980 2024-03-16T20:11:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f40bb950 con 0x7f02040c8980 2024-03-16T20:11:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020415a050 con 0x7f02040c8980 2024-03-16T20:11:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020414b450 con 0x7f02040c8980 2024-03-16T20:11:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:01.969+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020414b2b0 con 0x7f02040c8980 2024-03-16T20:11:01.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.009+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f01f40b9070 con 0x7f02040c8980 2024-03-16T20:11:01.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.009+0000 7f01faffd640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 0x7f01e8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:01.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.009+0000 7f020a237640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 0x7f01e8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:02.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.009+0000 7f020a237640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 0x7f01e8069350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0200001770 tx=0x7f0200009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:02.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.013+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f01f40efef0 con 0x7f02040c8980 2024-03-16T20:11:02.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.297+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f01f40ec190 con 0x7f02040c8980 2024-03-16T20:11:02.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.497+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f020414b2b0 con 0x7f02040c8980 2024-03-16T20:11:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f01faffd640 1 -- 172.21.15.32:0/2201753710 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f020414b2b0 con 0x7f02040c8980 2024-03-16T20:11:02.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:02.507 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:02.507 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 msgr2=0x7f01e8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 0x7f01e8069350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0200001770 tx=0x7f0200009040 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 msgr2=0x7f0204160c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f01f40ab040 tx=0x7f01f401adc0 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 shutdown_connections 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f01e8066e90 0x7f01e8069350 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0204151000 0x7f02041f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02040c8980 0x7f0204160c80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 --2- 172.21.15.32:0/2201753710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02040c36b0 0x7f0204160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 >> 172.21.15.32:0/2201753710 conn(0x7f020406c0d0 msgr2=0x7f0204140600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 shutdown_connections 2024-03-16T20:11:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:02.517+0000 7f020c49b640 1 -- 172.21.15.32:0/2201753710 wait complete. 2024-03-16T20:11:03.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:03.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:04.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.253+0000 7f81b0364640 1 Processor -- start 2024-03-16T20:11:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.253+0000 7f81b0364640 1 -- start start 2024-03-16T20:11:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81b0364640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f81a8071110 0x7f81a80c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81b0364640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80cae50 0x7f81a8151a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81b0364640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80ca1d0 con 0x7f81a80714f0 2024-03-16T20:11:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81b0364640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a80c34a0 con 0x7f81a8071110 2024-03-16T20:11:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81b0364640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a80c9ba0 con 0x7f81a80cae50 2024-03-16T20:11:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81ae100640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f81a8071110 0x7f81a80c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81ae901640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80cae50 0x7f81a8151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81ae100640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f81a8071110 0x7f81a80c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56950/0 (socket says 172.21.15.32:56950) 2024-03-16T20:11:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.257+0000 7f81ae100640 1 -- 172.21.15.32:0/285957623 learned_addr learned my addr 172.21.15.32:0/285957623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:04.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.261+0000 7f81ae100640 1 -- 172.21.15.32:0/285957623 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80cae50 msgr2=0x7f81a8151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:04.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.261+0000 7f81ae100640 1 --2- 172.21.15.32:0/285957623 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80cae50 0x7f81a8151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.261+0000 7f81ae100640 1 -- 172.21.15.32:0/285957623 >> v1:172.21.15.32:6789/0 conn(0x7f81a80714f0 legacy=0x7f81a80ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:04.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.261+0000 7f81ae100640 1 -- 172.21.15.32:0/285957623 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a80c2ee0 con 0x7f81a8071110 2024-03-16T20:11:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.265+0000 7f81ae100640 1 --2- 172.21.15.32:0/285957623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a80c82f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009870 tx=0x7f81a401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa5eb16db468ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:04.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81ad0fe640 1 -- 172.21.15.32:0/285957623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40a89d0 con 0x7f81a8071110 2024-03-16T20:11:04.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81ad0fe640 1 -- 172.21.15.32:0/285957623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a40a8b30 con 0x7f81a8071110 2024-03-16T20:11:04.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81ad0fe640 1 -- 172.21.15.32:0/285957623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40b0440 con 0x7f81a8071110 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 -- 172.21.15.32:0/285957623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 msgr2=0x7f81a80c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 --2- 172.21.15.32:0/285957623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a80c82f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009870 tx=0x7f81a401ed10 comp rx=0 tx=0).stop 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 -- 172.21.15.32:0/285957623 shutdown_connections 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 --2- 172.21.15.32:0/285957623 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80cae50 0x7f81a8151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 --2- 172.21.15.32:0/285957623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a80c82f0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 -- 172.21.15.32:0/285957623 >> 172.21.15.32:0/285957623 conn(0x7f81a806afc0 msgr2=0x7f81a806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:04.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 -- 172.21.15.32:0/285957623 shutdown_connections 2024-03-16T20:11:04.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.269+0000 7f81b0364640 1 -- 172.21.15.32:0/285957623 wait complete. 2024-03-16T20:11:04.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 Processor -- start 2024-03-16T20:11:04.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 -- start start 2024-03-16T20:11:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81a80714f0 0x7f81a81f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80c4fd0 0x7f81a814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a80cc3c0 con 0x7f81a8071110 2024-03-16T20:11:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a81f1d60 con 0x7f81a80714f0 2024-03-16T20:11:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81b0364640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a81f14c0 con 0x7f81a80c4fd0 2024-03-16T20:11:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81ae100640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81ad8ff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81a80714f0 0x7f81a81f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81ae100640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56962/0 (socket says 172.21.15.32:56962) 2024-03-16T20:11:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81ae100640 1 -- 172.21.15.32:0/751254642 learned_addr learned my addr 172.21.15.32:0/751254642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.273+0000 7f81ae901640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80c4fd0 0x7f81a814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.277+0000 7f81ae100640 1 -- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80c4fd0 msgr2=0x7f81a814c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.277+0000 7f81ae100640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80c4fd0 0x7f81a814c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.277+0000 7f81ae100640 1 -- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81a80714f0 msgr2=0x7f81a81f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.277+0000 7f81ae100640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81a80714f0 0x7f81a81f0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.277+0000 7f81ae100640 1 -- 172.21.15.32:0/751254642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40094d0 con 0x7f81a8071110 2024-03-16T20:11:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.281+0000 7f81ae100640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f81a401c940 tx=0x7f81a401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.285+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40b1d40 con 0x7f81a8071110 2024-03-16T20:11:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.285+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a40bb470 con 0x7f81a8071110 2024-03-16T20:11:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.285+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a81f17a0 con 0x7f81a8071110 2024-03-16T20:11:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.285+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40b1ea0 con 0x7f81a8071110 2024-03-16T20:11:04.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.285+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8157ca0 con 0x7f81a8071110 2024-03-16T20:11:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.289+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a8068000 con 0x7f81a8071110 2024-03-16T20:11:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.553+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f81a40b7070 con 0x7f81a8071110 2024-03-16T20:11:04.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.557+0000 7f819effd640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 0x7f818c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:04.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.557+0000 7f81ad8ff640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 0x7f818c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:04.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.557+0000 7f81ad8ff640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 0x7f818c069300 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f81a80cb8f0 tx=0x7f8198007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:04.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.569+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f81a40b5c80 con 0x7f81a8071110 2024-03-16T20:11:04.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.569+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f81a40f0350 con 0x7f81a8071110 2024-03-16T20:11:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.769+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81a8068000 con 0x7f81a8071110 2024-03-16T20:11:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f819effd640 1 -- 172.21.15.32:0/751254642 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f81a40b5030 con 0x7f81a8071110 2024-03-16T20:11:04.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:04.779 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:04.780 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 msgr2=0x7f818c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 0x7f818c069300 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f81a80cb8f0 tx=0x7f8198007e40 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 msgr2=0x7f81a81ed840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f81a401c940 tx=0x7f81a401bab0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 shutdown_connections 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f818c066e40 0x7f818c069300 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81a80c4fd0 0x7f81a814c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81a80714f0 0x7f81a81f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 --2- 172.21.15.32:0/751254642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a8071110 0x7f81a81ed840 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.789+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 >> 172.21.15.32:0/751254642 conn(0x7f81a806afc0 msgr2=0x7f81a8154ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:04.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.793+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 shutdown_connections 2024-03-16T20:11:04.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:04.793+0000 7f81b0364640 1 -- 172.21.15.32:0/751254642 wait complete. 2024-03-16T20:11:05.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:05.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:06.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 Processor -- start 2024-03-16T20:11:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 -- start start 2024-03-16T20:11:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcf0150d20 0x7efcf0156e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcf00c7990 0x7efcf015acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf015a840 con 0x7efcf0151100 2024-03-16T20:11:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf00c41f0 con 0x7efcf00c7990 2024-03-16T20:11:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.521+0000 7efcf5a8c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf0071280 con 0x7efcf0150d20 2024-03-16T20:11:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.525+0000 7efceeffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcf00c7990 0x7efcf015acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.525+0000 7efceeffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcf00c7990 0x7efcf015acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56972/0 (socket says 172.21.15.32:56972) 2024-03-16T20:11:06.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.525+0000 7efceeffd640 1 -- 172.21.15.32:0/1774755877 learned_addr learned my addr 172.21.15.32:0/1774755877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.525+0000 7efcef7fe640 1 --2- 172.21.15.32:0/1774755877 >> v2:172.21.15.59:3300/0 conn(0x7efcf0150d20 0x7efcf0156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.529+0000 7efceeffd640 1 -- 172.21.15.32:0/1774755877 >> v2:172.21.15.59:3300/0 conn(0x7efcf0150d20 msgr2=0x7efcf0156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.529+0000 7efceeffd640 1 --2- 172.21.15.32:0/1774755877 >> v2:172.21.15.59:3300/0 conn(0x7efcf0150d20 0x7efcf0156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.529+0000 7efceeffd640 1 -- 172.21.15.32:0/1774755877 >> v1:172.21.15.32:6789/0 conn(0x7efcf0151100 legacy=0x7efcf00c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.529+0000 7efceeffd640 1 -- 172.21.15.32:0/1774755877 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf0157d90 con 0x7efcf00c7990 2024-03-16T20:11:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.533+0000 7efceeffd640 1 --2- 172.21.15.32:0/1774755877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf00c7990 0x7efcf015acd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efcf0070b10 tx=0x7efce4020f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b58650f8380f75 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcee7fc640 1 -- 172.21.15.32:0/1774755877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40aac20 con 0x7efcf00c7990 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcee7fc640 1 -- 172.21.15.32:0/1774755877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40aad80 con 0x7efcf00c7990 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcee7fc640 1 -- 172.21.15.32:0/1774755877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40b2660 con 0x7efcf00c7990 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1774755877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf00c7990 msgr2=0x7efcf015acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1774755877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf00c7990 0x7efcf015acd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efcf0070b10 tx=0x7efce4020f20 comp rx=0 tx=0).stop 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1774755877 shutdown_connections 2024-03-16T20:11:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1774755877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf00c7990 0x7efcf015acd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1774755877 >> v2:172.21.15.59:3300/0 conn(0x7efcf0150d20 0x7efcf0156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1774755877 >> 172.21.15.32:0/1774755877 conn(0x7efcf00cb470 msgr2=0x7efcf00cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1774755877 shutdown_connections 2024-03-16T20:11:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.537+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1774755877 wait complete. 2024-03-16T20:11:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 Processor -- start 2024-03-16T20:11:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 -- start start 2024-03-16T20:11:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf00c7990 0x7efcf01ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcf0151100 0x7efcf0151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf01efe50 con 0x7efcf0150d20 2024-03-16T20:11:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf01eba80 con 0x7efcf00c7990 2024-03-16T20:11:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efcf5a8c640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf01ebd20 con 0x7efcf0151100 2024-03-16T20:11:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efceeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efceeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56978/0 (socket says 172.21.15.32:56978) 2024-03-16T20:11:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.541+0000 7efceeffd640 1 -- 172.21.15.32:0/1555812890 learned_addr learned my addr 172.21.15.32:0/1555812890 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efcef7fe640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf00c7990 0x7efcf01ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceffff640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcf0151100 0x7efcf0151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceeffd640 1 -- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcf0151100 msgr2=0x7efcf0151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceeffd640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcf0151100 0x7efcf0151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceeffd640 1 -- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf00c7990 msgr2=0x7efcf01ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceeffd640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf00c7990 0x7efcf01ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.545+0000 7efceeffd640 1 -- 172.21.15.32:0/1555812890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40094d0 con 0x7efcf0150d20 2024-03-16T20:11:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.549+0000 7efceeffd640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efce401db30 tx=0x7efce401dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:06.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.553+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40bc600 con 0x7efcf0150d20 2024-03-16T20:11:06.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.553+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40bc760 con 0x7efcf0150d20 2024-03-16T20:11:06.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.553+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf01ec000 con 0x7efcf0150d20 2024-03-16T20:11:06.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.553+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40bcb10 con 0x7efcf0150d20 2024-03-16T20:11:06.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.553+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf0152480 con 0x7efcf0150d20 2024-03-16T20:11:06.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.557+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf01ec4b0 con 0x7efcf0150d20 2024-03-16T20:11:06.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.581+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efce40bcc70 con 0x7efcf0150d20 2024-03-16T20:11:06.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.585+0000 7efcd7fff640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 0x7efcdc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:06.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.585+0000 7efcef7fe640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 0x7efcdc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:06.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.585+0000 7efcef7fe640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 0x7efcdc069380 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efce0009b60 tx=0x7efce0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.889+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efce40f6490 con 0x7efcf0150d20 2024-03-16T20:11:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:06.889+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efce40f2730 con 0x7efcf0150d20 2024-03-16T20:11:07.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.233+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcf01ec4b0 con 0x7efcf0150d20 2024-03-16T20:11:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.253+0000 7efcd7fff640 1 -- 172.21.15.32:0/1555812890 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7efcf01ec4b0 con 0x7efcf0150d20 2024-03-16T20:11:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:07.243 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 msgr2=0x7efcdc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 0x7efcdc069380 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efce0009b60 tx=0x7efce0009190 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 msgr2=0x7efcf01592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efce401db30 tx=0x7efce401dcc0 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 shutdown_connections 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcdc066ec0 0x7efcdc069380 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcf0151100 0x7efcf0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf0150d20 0x7efcf01592b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 --2- 172.21.15.32:0/1555812890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf00c7990 0x7efcf01ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 >> 172.21.15.32:0/1555812890 conn(0x7efcf00cb470 msgr2=0x7efcf00c5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 shutdown_connections 2024-03-16T20:11:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:07.257+0000 7efcf5a8c640 1 -- 172.21.15.32:0/1555812890 wait complete. 2024-03-16T20:11:07.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:07.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:08.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.993+0000 7f6a57617640 1 Processor -- start 2024-03-16T20:11:08.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.993+0000 7f6a57617640 1 -- start start 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a57617640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a501540c0 0x7f6a5015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a57617640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a500c3570 0x7f6a500c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a57617640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5006c460 con 0x7f6a5015ac10 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a57617640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a500c3f40 con 0x7f6a500c3570 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a57617640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a5015a240 con 0x7f6a501540c0 2024-03-16T20:11:08.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a55bb4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a500c3570 0x7f6a500c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a54bb2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6a5015ac10 0x7f6a500c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36814/0 (socket says 172.21.15.32:36814) 2024-03-16T20:11:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a54bb2640 1 -- 172.21.15.32:0/4009813019 learned_addr learned my addr 172.21.15.32:0/4009813019 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:08.997+0000 7f6a553b3640 1 --2- 172.21.15.32:0/4009813019 >> v2:172.21.15.59:3300/0 conn(0x7f6a501540c0 0x7f6a5015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.005+0000 7f6a55bb4640 1 -- 172.21.15.32:0/4009813019 >> v2:172.21.15.59:3300/0 conn(0x7f6a501540c0 msgr2=0x7f6a5015a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.005+0000 7f6a55bb4640 1 --2- 172.21.15.32:0/4009813019 >> v2:172.21.15.59:3300/0 conn(0x7f6a501540c0 0x7f6a5015a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.005+0000 7f6a55bb4640 1 -- 172.21.15.32:0/4009813019 >> v1:172.21.15.32:6789/0 conn(0x7f6a5015ac10 legacy=0x7f6a500c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:08.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.005+0000 7f6a55bb4640 1 -- 172.21.15.32:0/4009813019 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5006cf20 con 0x7f6a500c3570 2024-03-16T20:11:09.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.009+0000 7f6a55bb4640 1 --2- 172.21.15.32:0/4009813019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a500c3570 0x7f6a500c3a00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6a500724f0 tx=0x7f6a4801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75d6340eacc50c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a47fff640 1 -- 172.21.15.32:0/4009813019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a480a59c0 con 0x7f6a500c3570 2024-03-16T20:11:09.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a47fff640 1 -- 172.21.15.32:0/4009813019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a480a5b20 con 0x7f6a500c3570 2024-03-16T20:11:09.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a47fff640 1 -- 172.21.15.32:0/4009813019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a480ad440 con 0x7f6a500c3570 2024-03-16T20:11:09.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 -- 172.21.15.32:0/4009813019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a500c3570 msgr2=0x7f6a500c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 --2- 172.21.15.32:0/4009813019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a500c3570 0x7f6a500c3a00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6a500724f0 tx=0x7f6a4801c900 comp rx=0 tx=0).stop 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 -- 172.21.15.32:0/4009813019 shutdown_connections 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 --2- 172.21.15.32:0/4009813019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a500c3570 0x7f6a500c3a00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 --2- 172.21.15.32:0/4009813019 >> v2:172.21.15.59:3300/0 conn(0x7f6a501540c0 0x7f6a5015a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 -- 172.21.15.32:0/4009813019 >> 172.21.15.32:0/4009813019 conn(0x7f6a50070f90 msgr2=0x7f6a500713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 -- 172.21.15.32:0/4009813019 shutdown_connections 2024-03-16T20:11:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.013+0000 7f6a57617640 1 -- 172.21.15.32:0/4009813019 wait complete. 2024-03-16T20:11:09.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 Processor -- start 2024-03-16T20:11:09.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 -- start start 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a500c3570 0x7f6a5014c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a5015ac10 0x7f6a501462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501f6470 con 0x7f6a501540c0 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501558b0 con 0x7f6a500c3570 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a57617640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5014d810 con 0x7f6a5015ac10 2024-03-16T20:11:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a54bb2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a54bb2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57008/0 (socket says 172.21.15.32:57008) 2024-03-16T20:11:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.017+0000 7f6a54bb2640 1 -- 172.21.15.32:0/114276607 learned_addr learned my addr 172.21.15.32:0/114276607 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.021+0000 7f6a553b3640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a500c3570 0x7f6a5014c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 -- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a5015ac10 msgr2=0x7f6a501462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a5015ac10 0x7f6a501462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 -- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a500c3570 msgr2=0x7f6a5014c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a500c3570 0x7f6a5014c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 -- 172.21.15.32:0/114276607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48009a70 con 0x7f6a501540c0 2024-03-16T20:11:09.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.025+0000 7f6a54bb2640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000c400 tx=0x7f6a4000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:09.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40005cf0 con 0x7f6a501540c0 2024-03-16T20:11:09.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40005e50 con 0x7f6a501540c0 2024-03-16T20:11:09.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a400072d0 con 0x7f6a501540c0 2024-03-16T20:11:09.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a50154ab0 con 0x7f6a501540c0 2024-03-16T20:11:09.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a501f5db0 con 0x7f6a501540c0 2024-03-16T20:11:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.033+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a50070390 con 0x7f6a501540c0 2024-03-16T20:11:09.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.061+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6a40007430 con 0x7f6a501540c0 2024-03-16T20:11:09.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.061+0000 7f6a45ffb640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 0x7f6a34069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:09.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.061+0000 7f6a553b3640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 0x7f6a34069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:09.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.061+0000 7f6a553b3640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 0x7f6a34069300 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009b60 tx=0x7f6a4c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:09.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.369+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f6a40054f60 con 0x7f6a501540c0 2024-03-16T20:11:09.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.369+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f6a40051200 con 0x7f6a501540c0 2024-03-16T20:11:09.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.577+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a501f65b0 con 0x7f6a501540c0 2024-03-16T20:11:09.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.649+0000 7f6a45ffb640 1 -- 172.21.15.32:0/114276607 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f6a501f65b0 con 0x7f6a501540c0 2024-03-16T20:11:09.642 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:09.642 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:09.642 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:09.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 msgr2=0x7f6a34069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 0x7f6a34069300 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009b60 tx=0x7f6a4c009190 comp rx=0 tx=0).stop 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 msgr2=0x7f6a501f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000c400 tx=0x7f6a4000c8c0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 shutdown_connections 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f6a34066e40 0x7f6a34069300 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6a5015ac10 0x7f6a501462d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a501540c0 0x7f6a501f24c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 --2- 172.21.15.32:0/114276607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a500c3570 0x7f6a5014c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:09.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 >> 172.21.15.32:0/114276607 conn(0x7f6a50070f90 msgr2=0x7f6a500c90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:09.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 shutdown_connections 2024-03-16T20:11:09.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:09.653+0000 7f6a57617640 1 -- 172.21.15.32:0/114276607 wait complete. 2024-03-16T20:11:10.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:10.272 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:11.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:11.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 Processor -- start 2024-03-16T20:11:11.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 -- start start 2024-03-16T20:11:11.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f146815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14680c8bb0 0x7f146806f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f146815e250 0x7f146806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146815d5a0 con 0x7f14680c8bb0 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146815d880 con 0x7f146815e250 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f146d398640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146806ba40 con 0x7f14681543f0 2024-03-16T20:11:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f14667fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14680c8bb0 0x7f146806f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f14667fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14680c8bb0 0x7f146806f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38244/0 (socket says 172.21.15.32:38244) 2024-03-16T20:11:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f14667fc640 1 -- 172.21.15.32:0/3634595845 learned_addr learned my addr 172.21.15.32:0/3634595845 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f14677fe640 1 --2- 172.21.15.32:0/3634595845 >> v2:172.21.15.59:3300/0 conn(0x7f146815e250 0x7f146806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.389+0000 7f1466ffd640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f146815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.393+0000 7f14667fc640 1 -- 172.21.15.32:0/3634595845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 msgr2=0x7f146815dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.393+0000 7f14667fc640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f146815dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.393+0000 7f14667fc640 1 -- 172.21.15.32:0/3634595845 >> v2:172.21.15.59:3300/0 conn(0x7f146815e250 msgr2=0x7f146806fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.393+0000 7f14667fc640 1 --2- 172.21.15.32:0/3634595845 >> v2:172.21.15.59:3300/0 conn(0x7f146815e250 0x7f146806fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.393+0000 7f14667fc640 1 -- 172.21.15.32:0/3634595845 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146815ae70 con 0x7f14680c8bb0 2024-03-16T20:11:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.397+0000 7f14667fc640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14680c8bb0 0x7f146806f950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f14680c7e00 tx=0x7f145c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae16b3cdf0b3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f1465ffb640 1 -- 172.21.15.32:0/3634595845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0a89d0 con 0x7f14680c8bb0 2024-03-16T20:11:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f1465ffb640 1 -- 172.21.15.32:0/3634595845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0a8b30 con 0x7f14680c8bb0 2024-03-16T20:11:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f1465ffb640 1 -- 172.21.15.32:0/3634595845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0b0440 con 0x7f14680c8bb0 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 -- 172.21.15.32:0/3634595845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14680c8bb0 msgr2=0x7f146806f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14680c8bb0 0x7f146806f950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f14680c7e00 tx=0x7f145c01ed10 comp rx=0 tx=0).stop 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 -- 172.21.15.32:0/3634595845 shutdown_connections 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 --2- 172.21.15.32:0/3634595845 >> v2:172.21.15.59:3300/0 conn(0x7f146815e250 0x7f146806fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14680c8bb0 0x7f146806f950 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 --2- 172.21.15.32:0/3634595845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f146815dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 -- 172.21.15.32:0/3634595845 >> 172.21.15.32:0/3634595845 conn(0x7f14680cc7a0 msgr2=0x7f14680ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 -- 172.21.15.32:0/3634595845 shutdown_connections 2024-03-16T20:11:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.401+0000 7f146d398640 1 -- 172.21.15.32:0/3634595845 wait complete. 2024-03-16T20:11:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 Processor -- start 2024-03-16T20:11:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 -- start start 2024-03-16T20:11:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680c8bb0 0x7f146815cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f1468072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14681523b0 con 0x7f146815e250 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1468072120 con 0x7f14680c8bb0 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f146d398640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1468159e00 con 0x7f14681543f0 2024-03-16T20:11:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f14677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f14677fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38246/0 (socket says 172.21.15.32:38246) 2024-03-16T20:11:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f14677fe640 1 -- 172.21.15.32:0/522135062 learned_addr learned my addr 172.21.15.32:0/522135062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f14667fc640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f1468072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.405+0000 7f1466ffd640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680c8bb0 0x7f146815cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.409+0000 7f14677fe640 1 -- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 msgr2=0x7f1468072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.409+0000 7f14677fe640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f1468072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.409+0000 7f14677fe640 1 -- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680c8bb0 msgr2=0x7f146815cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.409+0000 7f14677fe640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680c8bb0 0x7f146815cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.409+0000 7f14677fe640 1 -- 172.21.15.32:0/522135062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0094d0 con 0x7f146815e250 2024-03-16T20:11:11.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.413+0000 7f14677fe640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1460007d40 tx=0x7f146000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:11.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460014420 con 0x7f146815e250 2024-03-16T20:11:11.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14600145a0 con 0x7f146815e250 2024-03-16T20:11:11.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14681511d0 con 0x7f146815e250 2024-03-16T20:11:11.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460014950 con 0x7f146815e250 2024-03-16T20:11:11.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1468151790 con 0x7f146815e250 2024-03-16T20:11:11.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.421+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1468072290 con 0x7f146815e250 2024-03-16T20:11:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.709+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1460014ab0 con 0x7f146815e250 2024-03-16T20:11:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.709+0000 7f144f7fe640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 0x7f1454069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.709+0000 7f1466ffd640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 0x7f1454069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.709+0000 7f1466ffd640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 0x7f1454069300 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f14580045f0 tx=0x7f1458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:11.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.733+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1460056b60 con 0x7f146815e250 2024-03-16T20:11:11.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.733+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1460052e00 con 0x7f146815e250 2024-03-16T20:11:11.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:11.945+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1468072290 con 0x7f146815e250 2024-03-16T20:11:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.009+0000 7f144f7fe640 1 -- 172.21.15.32:0/522135062 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1468072290 con 0x7f146815e250 2024-03-16T20:11:12.002 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:12.003 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 msgr2=0x7f1454069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 0x7f1454069300 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f14580045f0 tx=0x7f1458008040 comp rx=0 tx=0).stop 2024-03-16T20:11:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 msgr2=0x7f1468149d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1460007d40 tx=0x7f146000e4b0 comp rx=0 tx=0).stop 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 shutdown_connections 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1454066e40 0x7f1454069300 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146815e250 0x7f1468149d50 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f14681543f0 0x7f1468072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 --2- 172.21.15.32:0/522135062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680c8bb0 0x7f146815cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.013+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 >> 172.21.15.32:0/522135062 conn(0x7f14680cc7a0 msgr2=0x7f14680cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.017+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 shutdown_connections 2024-03-16T20:11:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:12.017+0000 7f146d398640 1 -- 172.21.15.32:0/522135062 wait complete. 2024-03-16T20:11:12.629 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:12.630 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:13.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:11:13.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:11:13.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 Processor -- start 2024-03-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 -- start start 2024-03-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ffc0c97b0 0x7f7ffc0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc158c80 con 0x7f7ffc0c8720 2024-03-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc0b0ad0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.749+0000 7f8001949640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc157cb0 con 0x7f7ffc151ef0 2024-03-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.753+0000 7f7ffaffd640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.753+0000 7f7ffa7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ffc0c97b0 0x7f7ffc0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.753+0000 7f7ffb7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7ffc0c8720 0x7f7ffc15b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:57486/0 (socket says 172.21.15.32:57486) 2024-03-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.753+0000 7f7ffb7fe640 1 -- 172.21.15.32:0/2894271770 learned_addr learned my addr 172.21.15.32:0/2894271770 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.757+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 msgr2=0x7f7ffc0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.757+0000 7f7ffa7fc640 1 --2- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.757+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2894271770 >> v1:172.21.15.59:6789/0 conn(0x7f7ffc0c8720 legacy=0x7f7ffc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.757+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2894271770 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc15bfc0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.765+0000 7f7ffa7fc640 1 --2- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc0c81e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097c0 tx=0x7f7ff001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7f3f0a520ab5c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f7ff9ffb640 1 -- 172.21.15.32:0/2894271770 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00a8990 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f7ff9ffb640 1 -- 172.21.15.32:0/2894271770 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff00a8af0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f7ff9ffb640 1 -- 172.21.15.32:0/2894271770 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00a8e80 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 -- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c97b0 msgr2=0x7f7ffc0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 --2- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc0c81e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097c0 tx=0x7f7ff001ec90 comp rx=0 tx=0).stop 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 -- 172.21.15.32:0/2894271770 shutdown_connections 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 --2- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc0c81e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 --2- 172.21.15.32:0/2894271770 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 -- 172.21.15.32:0/2894271770 >> 172.21.15.32:0/2894271770 conn(0x7f7ffc070f90 msgr2=0x7f7ffc0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 -- 172.21.15.32:0/2894271770 shutdown_connections 2024-03-16T20:11:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.769+0000 7f8001949640 1 -- 172.21.15.32:0/2894271770 wait complete. 2024-03-16T20:11:13.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 Processor -- start 2024-03-16T20:11:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 -- start start 2024-03-16T20:11:13.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c8720 0x7f7ffc1494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:13.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc15aef0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc149a10 con 0x7f7ffc0c8720 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f8001949640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc158fc0 con 0x7f7ffc151ef0 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f7ffa7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f7ffa7fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38262/0 (socket says 172.21.15.32:38262) 2024-03-16T20:11:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2150923600 learned_addr learned my addr 172.21.15.32:0/2150923600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f7ffb7fe640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.773+0000 7f7ffaffd640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c8720 0x7f7ffc1494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.781+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 msgr2=0x7f7ffc0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.781+0000 7f7ffa7fc640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.781+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c8720 msgr2=0x7f7ffc1494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.781+0000 7f7ffa7fc640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c8720 0x7f7ffc1494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.781+0000 7f7ffa7fc640 1 -- 172.21.15.32:0/2150923600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0009420 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.785+0000 7f7ffa7fc640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7ff001b8a0 tx=0x7f7ff001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00b1dc0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff00b9410 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc0cc9a0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00b97c0 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc06d530 con 0x7f7ffc0c97b0 2024-03-16T20:11:13.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:13.789+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc068000 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.084+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7ff00b7040 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.088+0000 7f7fe3fff640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 0x7f7fe8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.088+0000 7f7ffaffd640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 0x7f7fe8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.088+0000 7f7ffaffd640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 0x7f7fe8069380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009ac0 tx=0x7f7fec009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.112+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7ff00f3d60 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.112+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7ff00f0000 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.348+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ffc06f510 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.364+0000 7f7fe3fff640 1 -- 172.21.15.32:0/2150923600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f7ff00b5090 con 0x7f7ffc0c97b0 2024-03-16T20:11:14.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:14.357 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:14.357 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.364+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 msgr2=0x7f7fe8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 0x7f7fe8069380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009ac0 tx=0x7f7fec009190 comp rx=0 tx=0).stop 2024-03-16T20:11:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 msgr2=0x7f7ffc15b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7ff001b8a0 tx=0x7f7ff001ba30 comp rx=0 tx=0).stop 2024-03-16T20:11:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 shutdown_connections 2024-03-16T20:11:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7fe8066ec0 0x7f7fe8069380 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ffc151ef0 0x7f7ffc0cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ffc0c97b0 0x7f7ffc15b7a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 --2- 172.21.15.32:0/2150923600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0c8720 0x7f7ffc1494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 >> 172.21.15.32:0/2150923600 conn(0x7f7ffc070f90 msgr2=0x7f7ffc150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 shutdown_connections 2024-03-16T20:11:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:14.368+0000 7f8001949640 1 -- 172.21.15.32:0/2150923600 wait complete. 2024-03-16T20:11:14.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:14.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:15.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.096+0000 7f3efbab4640 1 Processor -- start 2024-03-16T20:11:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.096+0000 7f3efbab4640 1 -- start start 2024-03-16T20:11:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efbab4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef4150d70 0x7f3ef4073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efbab4640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ef40725e0 0x7f3ef4070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efbab4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef4072dd0 con 0x7f3ef40708a0 2024-03-16T20:11:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efbab4640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef415d790 con 0x7f3ef40725e0 2024-03-16T20:11:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efbab4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415da50 con 0x7f3ef4150d70 2024-03-16T20:11:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3ef904f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ef40725e0 0x7f3ef4070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3ef9850640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef4150d70 0x7f3ef4073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efa051640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3ef40708a0 0x7f3ef415a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:57496/0 (socket says 172.21.15.32:57496) 2024-03-16T20:11:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.100+0000 7f3efa051640 1 -- 172.21.15.32:0/3635247274 learned_addr learned my addr 172.21.15.32:0/3635247274 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.104+0000 7f3ef904f640 1 -- 172.21.15.32:0/3635247274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef4150d70 msgr2=0x7f3ef4073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.104+0000 7f3ef904f640 1 --2- 172.21.15.32:0/3635247274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef4150d70 0x7f3ef4073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.104+0000 7f3ef904f640 1 -- 172.21.15.32:0/3635247274 >> v1:172.21.15.59:6789/0 conn(0x7f3ef40708a0 legacy=0x7f3ef415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.104+0000 7f3ef904f640 1 -- 172.21.15.32:0/3635247274 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef415dd30 con 0x7f3ef40725e0 2024-03-16T20:11:16.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.108+0000 7f3ef904f640 1 --2- 172.21.15.32:0/3635247274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef40725e0 0x7f3ef4070360 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40097c0 tx=0x7f3ee401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81696d565b6eed25 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:16.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3ef884e640 1 -- 172.21.15.32:0/3635247274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40ab610 con 0x7f3ef40725e0 2024-03-16T20:11:16.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3ef884e640 1 -- 172.21.15.32:0/3635247274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee40ab770 con 0x7f3ef40725e0 2024-03-16T20:11:16.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3ef884e640 1 -- 172.21.15.32:0/3635247274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40abb00 con 0x7f3ef40725e0 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 -- 172.21.15.32:0/3635247274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef40725e0 msgr2=0x7f3ef4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 --2- 172.21.15.32:0/3635247274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef40725e0 0x7f3ef4070360 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40097c0 tx=0x7f3ee401c410 comp rx=0 tx=0).stop 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 -- 172.21.15.32:0/3635247274 shutdown_connections 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 --2- 172.21.15.32:0/3635247274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef40725e0 0x7f3ef4070360 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 --2- 172.21.15.32:0/3635247274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef4150d70 0x7f3ef4073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 -- 172.21.15.32:0/3635247274 >> 172.21.15.32:0/3635247274 conn(0x7f3ef406c0d0 msgr2=0x7f3ef406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 -- 172.21.15.32:0/3635247274 shutdown_connections 2024-03-16T20:11:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.112+0000 7f3efbab4640 1 -- 172.21.15.32:0/3635247274 wait complete. 2024-03-16T20:11:16.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 Processor -- start 2024-03-16T20:11:16.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 -- start start 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef40708a0 0x7f3ef4153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ef40725e0 0x7f3ef414a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef4143040 con 0x7f3ef4150d70 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef4148f10 con 0x7f3ef40725e0 2024-03-16T20:11:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efbab4640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef4149a30 con 0x7f3ef40708a0 2024-03-16T20:11:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efa051640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3ef904f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ef40725e0 0x7f3ef414a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efa051640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38282/0 (socket says 172.21.15.32:38282) 2024-03-16T20:11:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3efa051640 1 -- 172.21.15.32:0/1470046312 learned_addr learned my addr 172.21.15.32:0/1470046312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.116+0000 7f3ef9850640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef40708a0 0x7f3ef4153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 -- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef40708a0 msgr2=0x7f3ef4153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef40708a0 0x7f3ef4153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 -- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ef40725e0 msgr2=0x7f3ef414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ef40725e0 0x7f3ef414a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 -- 172.21.15.32:0/1470046312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee4009420 con 0x7f3ef4150d70 2024-03-16T20:11:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.124+0000 7f3efa051640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00caa0 tx=0x7f3eec00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:16.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.128+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec00ac00 con 0x7f3ef4150d70 2024-03-16T20:11:16.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.128+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eec00ad60 con 0x7f3ef4150d70 2024-03-16T20:11:16.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.132+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef415d560 con 0x7f3ef4150d70 2024-03-16T20:11:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.132+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec012620 con 0x7f3ef4150d70 2024-03-16T20:11:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.132+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef4149240 con 0x7f3ef4150d70 2024-03-16T20:11:16.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.132+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef4149870 con 0x7f3ef4150d70 2024-03-16T20:11:16.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.168+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3eec007070 con 0x7f3ef4150d70 2024-03-16T20:11:16.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.168+0000 7f3eea7fc640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 0x7f3ed8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:16.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.168+0000 7f3ef9850640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 0x7f3ed8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:16.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.168+0000 7f3ef9850640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 0x7f3ed8069300 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009ac0 tx=0x7f3ef0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:16.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.172+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f3eec0559a0 con 0x7f3ef4150d70 2024-03-16T20:11:16.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.464+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f3eec051c40 con 0x7f3ef4150d70 2024-03-16T20:11:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.672+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ef4068000 con 0x7f3ef4150d70 2024-03-16T20:11:16.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.684+0000 7f3eea7fc640 1 -- 172.21.15.32:0/1470046312 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f3eec017070 con 0x7f3ef4150d70 2024-03-16T20:11:16.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:16.679 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 msgr2=0x7f3ed8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 0x7f3ed8069300 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009ac0 tx=0x7f3ef0009190 comp rx=0 tx=0).stop 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 msgr2=0x7f3ef414a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00caa0 tx=0x7f3eec00cf60 comp rx=0 tx=0).stop 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 shutdown_connections 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3ed8066e40 0x7f3ed8069300 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ef4150d70 0x7f3ef414a810 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ef40725e0 0x7f3ef414a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 --2- 172.21.15.32:0/1470046312 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ef40708a0 0x7f3ef4153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:16.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 >> 172.21.15.32:0/1470046312 conn(0x7f3ef406c0d0 msgr2=0x7f3ef41597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:16.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 shutdown_connections 2024-03-16T20:11:16.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:16.688+0000 7f3efbab4640 1 -- 172.21.15.32:0/1470046312 wait complete. 2024-03-16T20:11:17.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:17.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:18.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 Processor -- start 2024-03-16T20:11:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 -- start start 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd938154300 0x7fd9380c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd938071a50 0x7fd938071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938072520 0x7fd93815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd938159fb0 con 0x7fd938154300 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9381546e0 con 0x7fd938071a50 2024-03-16T20:11:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd93d640640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380c7400 con 0x7fd938072520 2024-03-16T20:11:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd936ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd938154300 0x7fd9380c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd936ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd938154300 0x7fd9380c69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38298/0 (socket says 172.21.15.32:38298) 2024-03-16T20:11:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.420+0000 7fd936ffd640 1 -- 172.21.15.32:0/786268308 learned_addr learned my addr 172.21.15.32:0/786268308 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd9367fc640 1 --2- 172.21.15.32:0/786268308 >> v2:172.21.15.59:3300/0 conn(0x7fd938071a50 0x7fd938071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd9377fe640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938072520 0x7fd93815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd936ffd640 1 -- 172.21.15.32:0/786268308 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938072520 msgr2=0x7fd93815a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd936ffd640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938072520 0x7fd93815a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd936ffd640 1 -- 172.21.15.32:0/786268308 >> v2:172.21.15.59:3300/0 conn(0x7fd938071a50 msgr2=0x7fd938071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd936ffd640 1 --2- 172.21.15.32:0/786268308 >> v2:172.21.15.59:3300/0 conn(0x7fd938071a50 0x7fd938071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.424+0000 7fd936ffd640 1 -- 172.21.15.32:0/786268308 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380c76e0 con 0x7fd938154300 2024-03-16T20:11:18.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.428+0000 7fd936ffd640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938154300 0x7fd9380c69e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd938067530 tx=0x7fd92801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb11b548030923f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:18.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd935ffb640 1 -- 172.21.15.32:0/786268308 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280ad680 con 0x7fd938154300 2024-03-16T20:11:18.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd935ffb640 1 -- 172.21.15.32:0/786268308 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9280ad7e0 con 0x7fd938154300 2024-03-16T20:11:18.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd935ffb640 1 -- 172.21.15.32:0/786268308 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280adb70 con 0x7fd938154300 2024-03-16T20:11:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 -- 172.21.15.32:0/786268308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938154300 msgr2=0x7fd9380c69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938154300 0x7fd9380c69e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd938067530 tx=0x7fd92801c4f0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 -- 172.21.15.32:0/786268308 shutdown_connections 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938072520 0x7fd93815a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 --2- 172.21.15.32:0/786268308 >> v2:172.21.15.59:3300/0 conn(0x7fd938071a50 0x7fd938071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 --2- 172.21.15.32:0/786268308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938154300 0x7fd9380c69e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 -- 172.21.15.32:0/786268308 >> 172.21.15.32:0/786268308 conn(0x7fd9380cbfe0 msgr2=0x7fd9380cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 -- 172.21.15.32:0/786268308 shutdown_connections 2024-03-16T20:11:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.432+0000 7fd93d640640 1 -- 172.21.15.32:0/786268308 wait complete. 2024-03-16T20:11:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 Processor -- start 2024-03-16T20:11:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 -- start start 2024-03-16T20:11:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd938072520 0x7fd938146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938154300 0x7fd9381f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381f3300 con 0x7fd938071a50 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938147550 con 0x7fd938072520 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd93d640640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd938147830 con 0x7fd938154300 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd936ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd936ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38306/0 (socket says 172.21.15.32:38306) 2024-03-16T20:11:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd936ffd640 1 -- 172.21.15.32:0/1123124188 learned_addr learned my addr 172.21.15.32:0/1123124188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd9367fc640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd938072520 0x7fd938146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.436+0000 7fd9377fe640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938154300 0x7fd9381f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.440+0000 7fd936ffd640 1 -- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938154300 msgr2=0x7fd9381f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.440+0000 7fd936ffd640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938154300 0x7fd9381f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.440+0000 7fd936ffd640 1 -- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd938072520 msgr2=0x7fd938146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.440+0000 7fd936ffd640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd938072520 0x7fd938146ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.440+0000 7fd936ffd640 1 -- 172.21.15.32:0/1123124188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9280094d0 con 0x7fd938071a50 2024-03-16T20:11:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.444+0000 7fd936ffd640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd92801aba0 tx=0x7fd928019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280add60 con 0x7fd938071a50 2024-03-16T20:11:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9280b8470 con 0x7fd938071a50 2024-03-16T20:11:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280b8820 con 0x7fd938071a50 2024-03-16T20:11:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381f63d0 con 0x7fd938071a50 2024-03-16T20:11:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381f6990 con 0x7fd938071a50 2024-03-16T20:11:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.448+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9381f6b80 con 0x7fd938071a50 2024-03-16T20:11:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.484+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd9280b4070 con 0x7fd938071a50 2024-03-16T20:11:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.488+0000 7fd91f7fe640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 0x7fd924069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.488+0000 7fd9367fc640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 0x7fd924069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.488+0000 7fd9367fc640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 0x7fd924069380 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd9381538f0 tx=0x7fd92c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:18.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.764+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fd9280f0c20 con 0x7fd938071a50 2024-03-16T20:11:18.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.764+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fd9280ec7c0 con 0x7fd938071a50 2024-03-16T20:11:18.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:18.980+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd938068000 con 0x7fd938071a50 2024-03-16T20:11:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.080+0000 7fd91f7fe640 1 -- 172.21.15.32:0/1123124188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7fd9280efe80 con 0x7fd938071a50 2024-03-16T20:11:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:19.074 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 msgr2=0x7fd924069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 0x7fd924069380 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd9381538f0 tx=0x7fd92c008040 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 msgr2=0x7fd9381f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd92801aba0 tx=0x7fd928019290 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 shutdown_connections 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fd924066ec0 0x7fd924069380 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd938154300 0x7fd9381f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd938072520 0x7fd938146ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 --2- 172.21.15.32:0/1123124188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd938071a50 0x7fd9381f6eb0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 >> 172.21.15.32:0/1123124188 conn(0x7fd9380cbfe0 msgr2=0x7fd93806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 shutdown_connections 2024-03-16T20:11:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:19.084+0000 7fd93d640640 1 -- 172.21.15.32:0/1123124188 wait complete. 2024-03-16T20:11:19.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:19.703 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:20.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:20.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 Processor -- start 2024-03-16T20:11:20.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 -- start start 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f765c0c81e0 0x7f765c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f765c15b2a0 0x7f765c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c158c80 con 0x7f765c151ef0 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c0b0ad0 con 0x7f765c15b2a0 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.820+0000 7f76641b6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c157cb0 con 0x7f765c0c81e0 2024-03-16T20:11:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.824+0000 7f7662753640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f765c15b2a0 0x7f765c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.824+0000 7f7661751640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f765c0c81e0 0x7f765c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.824+0000 7f7662753640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f765c15b2a0 0x7f765c15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34792/0 (socket says 172.21.15.32:34792) 2024-03-16T20:11:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.824+0000 7f7662753640 1 -- 172.21.15.32:0/737705533 learned_addr learned my addr 172.21.15.32:0/737705533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:20.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.828+0000 7f7662753640 1 -- 172.21.15.32:0/737705533 >> v2:172.21.15.59:3300/0 conn(0x7f765c0c81e0 msgr2=0x7f765c0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:20.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.828+0000 7f7662753640 1 --2- 172.21.15.32:0/737705533 >> v2:172.21.15.59:3300/0 conn(0x7f765c0c81e0 0x7f765c0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.828+0000 7f7662753640 1 -- 172.21.15.32:0/737705533 >> v1:172.21.15.59:6789/0 conn(0x7f765c151ef0 legacy=0x7f765c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:20.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.828+0000 7f7662753640 1 -- 172.21.15.32:0/737705533 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c15bfc0 con 0x7f765c15b2a0 2024-03-16T20:11:20.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.828+0000 7f7662753640 1 --2- 172.21.15.32:0/737705533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c15b6a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f765c0cad10 tx=0x7f765401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146ae5f21a91a48c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:20.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f7660f50640 1 -- 172.21.15.32:0/737705533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540af400 con 0x7f765c15b2a0 2024-03-16T20:11:20.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f7660f50640 1 -- 172.21.15.32:0/737705533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76540ab050 con 0x7f765c15b2a0 2024-03-16T20:11:20.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f7660f50640 1 -- 172.21.15.32:0/737705533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540af780 con 0x7f765c15b2a0 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- 172.21.15.32:0/737705533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 msgr2=0x7f765c15b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 --2- 172.21.15.32:0/737705533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c15b6a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f765c0cad10 tx=0x7f765401f1a0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- 172.21.15.32:0/737705533 shutdown_connections 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 --2- 172.21.15.32:0/737705533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c15b6a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 --2- 172.21.15.32:0/737705533 >> v2:172.21.15.59:3300/0 conn(0x7f765c0c81e0 0x7f765c0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- 172.21.15.32:0/737705533 >> 172.21.15.32:0/737705533 conn(0x7f765c070f90 msgr2=0x7f765c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:20.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- 172.21.15.32:0/737705533 shutdown_connections 2024-03-16T20:11:20.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- 172.21.15.32:0/737705533 wait complete. 2024-03-16T20:11:20.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 Processor -- start 2024-03-16T20:11:20.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.836+0000 7f76641b6640 1 -- start start 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f765c0c81e0 0x7f765c1efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c15a770 con 0x7f765c15b2a0 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c15b060 con 0x7f765c151ef0 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f76641b6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c14c6b0 con 0x7f765c0c81e0 2024-03-16T20:11:20.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f7661751640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:20.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f7661f52640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f765c0c81e0 0x7f765c1efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:20.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f7662753640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:20.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f7661751640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:52332/0 (socket says 172.21.15.32:52332) 2024-03-16T20:11:20.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.840+0000 7f7661751640 1 -- 172.21.15.32:0/3084458849 learned_addr learned my addr 172.21.15.32:0/3084458849 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.844+0000 7f7661751640 1 -- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f765c0c81e0 msgr2=0x7f765c1efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.844+0000 7f7661751640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f765c0c81e0 0x7f765c1efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.844+0000 7f7661751640 1 -- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 msgr2=0x7f765c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.844+0000 7f7661751640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c0cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.844+0000 7f7661751640 1 -- 172.21.15.32:0/3084458849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009b30 con 0x7f765c151ef0 2024-03-16T20:11:20.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.848+0000 7f7661751640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f764c00e960 tx=0x7f764c00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:20.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.852+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c009690 con 0x7f765c151ef0 2024-03-16T20:11:20.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.852+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c06c850 con 0x7f765c151ef0 2024-03-16T20:11:20.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.852+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c15a1b0 con 0x7f765c151ef0 2024-03-16T20:11:20.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.856+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c068000 con 0x7f765c151ef0 2024-03-16T20:11:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:20.856+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c00ca00 con 0x7f765c151ef0 2024-03-16T20:11:21.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.192+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c00cd70 con 0x7f765c151ef0 2024-03-16T20:11:21.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.196+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f764c013470 con 0x7f765c151ef0 2024-03-16T20:11:21.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.196+0000 7f76527fc640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 0x7f7640069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:21.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.196+0000 7f7661f52640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 0x7f7640069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:21.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.196+0000 7f7661f52640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 0x7f7640069300 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7658007fd0 tx=0x7f7658009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:21.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.200+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f764c056000 con 0x7f765c151ef0 2024-03-16T20:11:21.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.220+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f764c0522a0 con 0x7f765c151ef0 2024-03-16T20:11:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.464+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f765c155610 con 0x7f765c151ef0 2024-03-16T20:11:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.624+0000 7f76527fc640 1 -- 172.21.15.32:0/3084458849 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f764c017090 con 0x7f765c151ef0 2024-03-16T20:11:21.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:21.619 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:21.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 msgr2=0x7f7640069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:21.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 0x7f7640069300 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7658007fd0 tx=0x7f7658009040 comp rx=0 tx=0).stop 2024-03-16T20:11:21.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 msgr2=0x7f765c15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:21.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f764c00e960 tx=0x7f764c00ee20 comp rx=0 tx=0).stop 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 shutdown_connections 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7640066e40 0x7f7640069300 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f765c15b2a0 0x7f765c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765c151ef0 0x7f765c15ab20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 --2- 172.21.15.32:0/3084458849 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f765c0c81e0 0x7f765c1efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 >> 172.21.15.32:0/3084458849 conn(0x7f765c070f90 msgr2=0x7f765c155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 shutdown_connections 2024-03-16T20:11:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:21.628+0000 7f76641b6640 1 -- 172.21.15.32:0/3084458849 wait complete. 2024-03-16T20:11:22.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:22.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:23.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:23.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.360+0000 7f77e0a34640 1 Processor -- start 2024-03-16T20:11:23.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.360+0000 7f77e0a34640 1 -- start start 2024-03-16T20:11:23.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77e0a34640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77dc0c7230 0x7f77dc06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:23.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77e0a34640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc0c5ce0 con 0x7f77dc073350 2024-03-16T20:11:23.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77e0a34640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc06afc0 con 0x7f77dc158b30 2024-03-16T20:11:23.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77e0a34640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77dc072f80 con 0x7f77dc0c7230 2024-03-16T20:11:23.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77da59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77dc0c7230 0x7f77dc06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77d9d9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77dc073350 0x7f77dc158420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57750/0 (socket says 172.21.15.32:57750) 2024-03-16T20:11:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.364+0000 7f77d9d9b640 1 -- 172.21.15.32:0/140914102 learned_addr learned my addr 172.21.15.32:0/140914102 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.368+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349638072 0 0) 0x7f77dc0c5ce0 con 0x7f77dc073350 2024-03-16T20:11:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.368+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c4003530 con 0x7f77dc073350 2024-03-16T20:11:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.372+0000 7f77da59c640 1 -- 172.21.15.32:0/140914102 >> v1:172.21.15.59:6789/0 conn(0x7f77dc158b30 legacy=0x7f77dc162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.372+0000 7f77da59c640 1 -- 172.21.15.32:0/140914102 >> v1:172.21.15.32:6789/0 conn(0x7f77dc073350 legacy=0x7f77dc158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.372+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650437925 0 0) 0x7f77c4003530 con 0x7f77dc073350 2024-03-16T20:11:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.372+0000 7f77da59c640 1 -- 172.21.15.32:0/140914102 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc0c5f30 con 0x7f77dc0c7230 2024-03-16T20:11:23.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.384+0000 7f77da59c640 1 --2- 172.21.15.32:0/140914102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc06b450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0097c0 tx=0x7f77cc01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=260e4aafd05a0f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0001680 con 0x7f77dc0c7230 2024-03-16T20:11:23.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc0a8630 con 0x7f77dc0c7230 2024-03-16T20:11:23.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77d959a640 1 -- 172.21.15.32:0/140914102 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc0a89c0 con 0x7f77dc0c7230 2024-03-16T20:11:23.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- 172.21.15.32:0/140914102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 msgr2=0x7f77dc06b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 --2- 172.21.15.32:0/140914102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc06b450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0097c0 tx=0x7f77cc01ebb0 comp rx=0 tx=0).stop 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- 172.21.15.32:0/140914102 shutdown_connections 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 --2- 172.21.15.32:0/140914102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc06b450 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- 172.21.15.32:0/140914102 >> 172.21.15.32:0/140914102 conn(0x7f77dc0cc7a0 msgr2=0x7f77dc0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- 172.21.15.32:0/140914102 shutdown_connections 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- 172.21.15.32:0/140914102 wait complete. 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 Processor -- start 2024-03-16T20:11:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- start start 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77dc073350 0x7f77dc1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc1e0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc13cc80 con 0x7f77dc158b30 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc0c3c00 con 0x7f77dc0c7230 2024-03-16T20:11:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.392+0000 7f77e0a34640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc0c3ee0 con 0x7f77dc073350 2024-03-16T20:11:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.396+0000 7f77dad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.396+0000 7f77da59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77dc073350 0x7f77dc1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.396+0000 7f77dad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34808/0 (socket says 172.21.15.32:34808) 2024-03-16T20:11:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.396+0000 7f77dad9d640 1 -- 172.21.15.32:0/1905849979 learned_addr learned my addr 172.21.15.32:0/1905849979 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:23.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.396+0000 7f77d9d9b640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc1e0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.400+0000 7f77dad9d640 1 -- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77dc073350 msgr2=0x7f77dc1f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.400+0000 7f77dad9d640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77dc073350 0x7f77dc1f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.400+0000 7f77dad9d640 1 -- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 msgr2=0x7f77dc1e0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.400+0000 7f77dad9d640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc1e0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.400+0000 7f77dad9d640 1 -- 172.21.15.32:0/1905849979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc009420 con 0x7f77dc158b30 2024-03-16T20:11:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.404+0000 7f77dad9d640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c730 tx=0x7f77d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4003b80 con 0x7f77dc158b30 2024-03-16T20:11:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d4003ce0 con 0x7f77dc158b30 2024-03-16T20:11:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc0c41c0 con 0x7f77dc158b30 2024-03-16T20:11:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4005310 con 0x7f77dc158b30 2024-03-16T20:11:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77dc0c4780 con 0x7f77dc158b30 2024-03-16T20:11:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.408+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77dc068000 con 0x7f77dc158b30 2024-03-16T20:11:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.440+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f77d400a640 con 0x7f77dc158b30 2024-03-16T20:11:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.444+0000 7f77caffd640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 0x7f77bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.444+0000 7f77da59c640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 0x7f77bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.444+0000 7f77da59c640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 0x7f77bc069300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0b2000 tx=0x7f77cc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.708+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f77d4055fe0 con 0x7f77dc158b30 2024-03-16T20:11:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:23.708+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f77d4052280 con 0x7f77dc158b30 2024-03-16T20:11:24.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.012+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77dc0cacd0 con 0x7f77dc158b30 2024-03-16T20:11:24.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.032+0000 7f77caffd640 1 -- 172.21.15.32:0/1905849979 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f77d4055940 con 0x7f77dc158b30 2024-03-16T20:11:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:24.026 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:24.026 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 msgr2=0x7f77bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 0x7f77bc069300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0b2000 tx=0x7f77cc0aa070 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 msgr2=0x7f77dc0c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c730 tx=0x7f77d400cbf0 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 shutdown_connections 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f77bc066e40 0x7f77bc069300 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77dc158b30 0x7f77dc0c4f80 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77dc0c7230 0x7f77dc1e0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 --2- 172.21.15.32:0/1905849979 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77dc073350 0x7f77dc1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 >> 172.21.15.32:0/1905849979 conn(0x7f77dc0cc7a0 msgr2=0x7f77dc1576c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:24.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 shutdown_connections 2024-03-16T20:11:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:24.036+0000 7f77e0a34640 1 -- 172.21.15.32:0/1905849979 wait complete. 2024-03-16T20:11:24.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:24.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:25.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 Processor -- start 2024-03-16T20:11:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 -- start start 2024-03-16T20:11:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b4c0c7470 0x7f1b4c165780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c070c90 con 0x7f1b4c070840 2024-03-16T20:11:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c0c8590 con 0x7f1b4c153fa0 2024-03-16T20:11:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b510ed640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c0c3f00 con 0x7f1b4c0c7470 2024-03-16T20:11:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b4b59e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b4c0c7470 0x7f1b4c165780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b4ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1b4c153fa0 0x7f1b4c15d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49590/0 (socket says 172.21.15.32:49590) 2024-03-16T20:11:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.764+0000 7f1b4ad9d640 1 -- 172.21.15.32:0/4023203629 learned_addr learned my addr 172.21.15.32:0/4023203629 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.768+0000 7f1b4b59e640 1 -- 172.21.15.32:0/4023203629 >> v1:172.21.15.59:6789/0 conn(0x7f1b4c153fa0 legacy=0x7f1b4c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.768+0000 7f1b4b59e640 1 -- 172.21.15.32:0/4023203629 >> v1:172.21.15.32:6789/0 conn(0x7f1b4c070840 legacy=0x7f1b4c0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.768+0000 7f1b4b59e640 1 -- 172.21.15.32:0/4023203629 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c073590 con 0x7f1b4c0c7470 2024-03-16T20:11:25.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.772+0000 7f1b4b59e640 1 --2- 172.21.15.32:0/4023203629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c165780 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009d90 tx=0x7f1b4401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b89ad4bc9b3fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b49d9b640 1 -- 172.21.15.32:0/4023203629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440af600 con 0x7f1b4c0c7470 2024-03-16T20:11:25.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b49d9b640 1 -- 172.21.15.32:0/4023203629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b440ab050 con 0x7f1b4c0c7470 2024-03-16T20:11:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b49d9b640 1 -- 172.21.15.32:0/4023203629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440af980 con 0x7f1b4c0c7470 2024-03-16T20:11:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b510ed640 1 -- 172.21.15.32:0/4023203629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c0c7470 msgr2=0x7f1b4c165780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b510ed640 1 --2- 172.21.15.32:0/4023203629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c165780 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009d90 tx=0x7f1b4401f150 comp rx=0 tx=0).stop 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.776+0000 7f1b510ed640 1 -- 172.21.15.32:0/4023203629 shutdown_connections 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 --2- 172.21.15.32:0/4023203629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c165780 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- 172.21.15.32:0/4023203629 >> 172.21.15.32:0/4023203629 conn(0x7f1b4c06c0d0 msgr2=0x7f1b4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- 172.21.15.32:0/4023203629 shutdown_connections 2024-03-16T20:11:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- 172.21.15.32:0/4023203629 wait complete. 2024-03-16T20:11:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 Processor -- start 2024-03-16T20:11:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- start start 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b4c153fa0 0x7f1b4c147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c15b020 con 0x7f1b4c070840 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c146950 con 0x7f1b4c0c7470 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b510ed640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0c3980 con 0x7f1b4c153fa0 2024-03-16T20:11:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.780+0000 7f1b4ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:25.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.784+0000 7f1b4ad9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34820/0 (socket says 172.21.15.32:34820) 2024-03-16T20:11:25.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.784+0000 7f1b4ad9d640 1 -- 172.21.15.32:0/27256063 learned_addr learned my addr 172.21.15.32:0/27256063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:25.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.784+0000 7f1b4a59c640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:25.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.784+0000 7f1b4b59e640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b4c153fa0 0x7f1b4c147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.788+0000 7f1b4ad9d640 1 -- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b4c153fa0 msgr2=0x7f1b4c147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.788+0000 7f1b4ad9d640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b4c153fa0 0x7f1b4c147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.788+0000 7f1b4ad9d640 1 -- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b4c0c7470 msgr2=0x7f1b4c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.788+0000 7f1b4ad9d640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:25.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.788+0000 7f1b4ad9d640 1 -- 172.21.15.32:0/27256063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44009a70 con 0x7f1b4c070840 2024-03-16T20:11:25.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.792+0000 7f1b4ad9d640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c370 tx=0x7f1b3c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c005cf0 con 0x7f1b4c070840 2024-03-16T20:11:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c005e50 con 0x7f1b4c070840 2024-03-16T20:11:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c1f0500 con 0x7f1b4c070840 2024-03-16T20:11:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c0072f0 con 0x7f1b4c070840 2024-03-16T20:11:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1f07a0 con 0x7f1b4c070840 2024-03-16T20:11:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.796+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c15ae20 con 0x7f1b4c070840 2024-03-16T20:11:25.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.832+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1b3c007450 con 0x7f1b4c070840 2024-03-16T20:11:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.832+0000 7f1b337fe640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 0x7f1b38069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.832+0000 7f1b4a59c640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 0x7f1b38069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:25.832+0000 7f1b4a59c640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 0x7f1b38069350 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0c3b90 tx=0x7f1b40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:26.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.144+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f1b3c054f60 con 0x7f1b4c070840 2024-03-16T20:11:26.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.144+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f1b3c051200 con 0x7f1b4c070840 2024-03-16T20:11:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.348+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4c1f0930 con 0x7f1b4c070840 2024-03-16T20:11:26.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.368+0000 7f1b337fe640 1 -- 172.21.15.32:0/27256063 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f1b3c017070 con 0x7f1b4c070840 2024-03-16T20:11:26.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:26.360 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:26.361 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 msgr2=0x7f1b38069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 0x7f1b38069350 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0c3b90 tx=0x7f1b40009040 comp rx=0 tx=0).stop 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 msgr2=0x7f1b4c146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c370 tx=0x7f1b3c00c830 comp rx=0 tx=0).stop 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 shutdown_connections 2024-03-16T20:11:26.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1b38066e90 0x7f1b38069350 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b4c153fa0 0x7f1b4c147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b4c0c7470 0x7f1b4c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 --2- 172.21.15.32:0/27256063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c070840 0x7f1b4c146410 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 >> 172.21.15.32:0/27256063 conn(0x7f1b4c06c0d0 msgr2=0x7f1b4c14e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 shutdown_connections 2024-03-16T20:11:26.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:26.372+0000 7f1b510ed640 1 -- 172.21.15.32:0/27256063 wait complete. 2024-03-16T20:11:26.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:26.983 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:27.865 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:11.943 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.870 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:11.948 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.893 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:11.909 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.896 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:11.912 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.959 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:11.975 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.961 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:11.977 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:27.985 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:12.063 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:27.989 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:12.005 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:28.065 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:12.080 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:11:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.096+0000 7f108253d640 1 Processor -- start 2024-03-16T20:11:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.096+0000 7f108253d640 1 -- start start 2024-03-16T20:11:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.100+0000 7f108253d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f107c071130 0x7f107c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.100+0000 7f108253d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c0c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.100+0000 7f108253d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0c2940 con 0x7f107c071510 2024-03-16T20:11:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.100+0000 7f108253d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c0c8a30 con 0x7f107c071130 2024-03-16T20:11:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.100+0000 7f108253d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0c9950 con 0x7f107c0c8fe0 2024-03-16T20:11:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.112+0000 7f107bfff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f107c071130 0x7f107c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:28.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.116+0000 7f107b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f107c071510 0x7f107c0c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:49602/0 (socket says 172.21.15.32:49602) 2024-03-16T20:11:28.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.116+0000 7f107b7fe640 1 -- 172.21.15.32:0/3832403520 learned_addr learned my addr 172.21.15.32:0/3832403520 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.120+0000 7f1080ada640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c0c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.140+0000 7f107bfff640 1 -- 172.21.15.32:0/3832403520 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 msgr2=0x7f107c0c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.140+0000 7f107bfff640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c0c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.140+0000 7f107bfff640 1 -- 172.21.15.32:0/3832403520 >> v1:172.21.15.59:6789/0 conn(0x7f107c071510 legacy=0x7f107c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.140+0000 7f107bfff640 1 -- 172.21.15.32:0/3832403520 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0b2de0 con 0x7f107c071130 2024-03-16T20:11:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.164+0000 7f107bfff640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c0c9b90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f106c0097c0 tx=0x7f106c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca121143a816c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f107affd640 1 -- 172.21.15.32:0/3832403520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0a8990 con 0x7f107c071130 2024-03-16T20:11:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f107affd640 1 -- 172.21.15.32:0/3832403520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c0a8af0 con 0x7f107c071130 2024-03-16T20:11:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f107affd640 1 -- 172.21.15.32:0/3832403520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0a8e80 con 0x7f107c071130 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 -- 172.21.15.32:0/3832403520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 msgr2=0x7f107c0c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c0c9b90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f106c0097c0 tx=0x7f106c01ec90 comp rx=0 tx=0).stop 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 -- 172.21.15.32:0/3832403520 shutdown_connections 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 --2- 172.21.15.32:0/3832403520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c0c9b90 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 -- 172.21.15.32:0/3832403520 >> 172.21.15.32:0/3832403520 conn(0x7f107c06b190 msgr2=0x7f107c06f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 -- 172.21.15.32:0/3832403520 shutdown_connections 2024-03-16T20:11:28.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.176+0000 7f108253d640 1 -- 172.21.15.32:0/3832403520 wait complete. 2024-03-16T20:11:28.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 Processor -- start 2024-03-16T20:11:28.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 -- start start 2024-03-16T20:11:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c071510 0x7f107c1eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0cdd70 con 0x7f107c071130 2024-03-16T20:11:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c157dc0 con 0x7f107c071510 2024-03-16T20:11:28.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.180+0000 7f108253d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0cd180 con 0x7f107c0c8fe0 2024-03-16T20:11:28.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.184+0000 7f107bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:28.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.184+0000 7f107bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34840/0 (socket says 172.21.15.32:34840) 2024-03-16T20:11:28.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.184+0000 7f107bfff640 1 -- 172.21.15.32:0/2575522074 learned_addr learned my addr 172.21.15.32:0/2575522074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.188+0000 7f107b7fe640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c071510 0x7f107c1eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:28.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.196+0000 7f107bfff640 1 -- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 msgr2=0x7f107c146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.196+0000 7f107bfff640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.196+0000 7f107bfff640 1 -- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c071510 msgr2=0x7f107c1eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.196+0000 7f107bfff640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c071510 0x7f107c1eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.196+0000 7f107bfff640 1 -- 172.21.15.32:0/2575522074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c009420 con 0x7f107c071130 2024-03-16T20:11:28.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.212+0000 7f107bfff640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f106c0b3040 tx=0x7f106c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0b0d90 con 0x7f107c071130 2024-03-16T20:11:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c0b9430 con 0x7f107c071130 2024-03-16T20:11:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0b97e0 con 0x7f107c071130 2024-03-16T20:11:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c0ce640 con 0x7f107c071130 2024-03-16T20:11:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c0cd6b0 con 0x7f107c071130 2024-03-16T20:11:28.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.224+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c068000 con 0x7f107c071130 2024-03-16T20:11:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.560+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f106c0b7040 con 0x7f107c071130 2024-03-16T20:11:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.560+0000 7f1078ff9640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 0x7f1068069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.560+0000 7f107b7fe640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 0x7f1068069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.560+0000 7f107b7fe640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 0x7f1068069350 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f107c0c97a0 tx=0x7f1070009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.576+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f106c0f4580 con 0x7f107c071130 2024-03-16T20:11:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.576+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f106c0f0120 con 0x7f107c071130 2024-03-16T20:11:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.820+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107c0b1ce0 con 0x7f107c071130 2024-03-16T20:11:28.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f1078ff9640 1 -- 172.21.15.32:0/2575522074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f106c0b5020 con 0x7f107c071130 2024-03-16T20:11:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:28.844 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 msgr2=0x7f1068069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 0x7f1068069350 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f107c0c97a0 tx=0x7f1070009040 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 msgr2=0x7f107c1ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f106c0b3040 tx=0x7f106c01ba30 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 shutdown_connections 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f1068066e90 0x7f1068069350 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f107c0c8fe0 0x7f107c146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c071510 0x7f107c1eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 --2- 172.21.15.32:0/2575522074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f107c071130 0x7f107c1ee890 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 >> 172.21.15.32:0/2575522074 conn(0x7f107c06b190 msgr2=0x7f107c150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 shutdown_connections 2024-03-16T20:11:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:28.852+0000 7f108253d640 1 -- 172.21.15.32:0/2575522074 wait complete. 2024-03-16T20:11:29.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:29.467 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:30.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 Processor -- start 2024-03-16T20:11:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 -- start start 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f86041540c0 0x7f860415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860415ac10 0x7f86040c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860406c460 con 0x7f86040c8720 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860415a240 con 0x7f860415ac10 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.580+0000 7f860adf3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860406cf20 con 0x7f86041540c0 2024-03-16T20:11:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.584+0000 7f8603fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860415ac10 0x7f86040c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.584+0000 7f8608b8f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f86041540c0 0x7f860415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.584+0000 7f8603fff640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860415ac10 0x7f86040c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48980/0 (socket says 172.21.15.32:48980) 2024-03-16T20:11:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.584+0000 7f8603fff640 1 -- 172.21.15.32:0/3457902660 learned_addr learned my addr 172.21.15.32:0/3457902660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.588+0000 7f8603fff640 1 -- 172.21.15.32:0/3457902660 >> v2:172.21.15.59:3300/0 conn(0x7f86041540c0 msgr2=0x7f860415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.588+0000 7f8603fff640 1 --2- 172.21.15.32:0/3457902660 >> v2:172.21.15.59:3300/0 conn(0x7f86041540c0 0x7f860415a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.588+0000 7f8603fff640 1 -- 172.21.15.32:0/3457902660 >> v1:172.21.15.32:6789/0 conn(0x7f86040c8720 legacy=0x7f86040c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.588+0000 7f8603fff640 1 -- 172.21.15.32:0/3457902660 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860406d200 con 0x7f860415ac10 2024-03-16T20:11:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.596+0000 7f8603fff640 1 --2- 172.21.15.32:0/3457902660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040c81e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f86040724f0 tx=0x7f85f001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed63bb96a7d602a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f86037fe640 1 -- 172.21.15.32:0/3457902660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00ab600 con 0x7f860415ac10 2024-03-16T20:11:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f86037fe640 1 -- 172.21.15.32:0/3457902660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f00ab760 con 0x7f860415ac10 2024-03-16T20:11:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f86037fe640 1 -- 172.21.15.32:0/3457902660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00abaf0 con 0x7f860415ac10 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 -- 172.21.15.32:0/3457902660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 msgr2=0x7f86040c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 --2- 172.21.15.32:0/3457902660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040c81e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f86040724f0 tx=0x7f85f001c4f0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 -- 172.21.15.32:0/3457902660 shutdown_connections 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 --2- 172.21.15.32:0/3457902660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040c81e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 --2- 172.21.15.32:0/3457902660 >> v2:172.21.15.59:3300/0 conn(0x7f86041540c0 0x7f860415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 -- 172.21.15.32:0/3457902660 >> 172.21.15.32:0/3457902660 conn(0x7f8604070f90 msgr2=0x7f86040713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 -- 172.21.15.32:0/3457902660 shutdown_connections 2024-03-16T20:11:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.600+0000 7f860adf3640 1 -- 172.21.15.32:0/3457902660 wait complete. 2024-03-16T20:11:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 Processor -- start 2024-03-16T20:11:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 -- start start 2024-03-16T20:11:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f86040c8720 0x7f8604143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86041540c0 0x7f86040cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604149c00 con 0x7f860415ac10 2024-03-16T20:11:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041f70e0 con 0x7f86041540c0 2024-03-16T20:11:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f860adf3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86041f6210 con 0x7f86040c8720 2024-03-16T20:11:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f8609390640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f8609390640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48984/0 (socket says 172.21.15.32:48984) 2024-03-16T20:11:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f8609390640 1 -- 172.21.15.32:0/1176717645 learned_addr learned my addr 172.21.15.32:0/1176717645 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f8603fff640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86041540c0 0x7f86040cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.604+0000 7f8608b8f640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f86040c8720 0x7f8604143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.608+0000 7f8609390640 1 -- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f86040c8720 msgr2=0x7f8604143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.608+0000 7f8609390640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f86040c8720 0x7f8604143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.608+0000 7f8609390640 1 -- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86041540c0 msgr2=0x7f86040cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.608+0000 7f8609390640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86041540c0 0x7f86040cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.608+0000 7f8609390640 1 -- 172.21.15.32:0/1176717645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00094d0 con 0x7f860415ac10 2024-03-16T20:11:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.612+0000 7f8609390640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00c730 tx=0x7f85fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc003b80 con 0x7f860415ac10 2024-03-16T20:11:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc003ce0 con 0x7f860415ac10 2024-03-16T20:11:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041f64d0 con 0x7f860415ac10 2024-03-16T20:11:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00aa30 con 0x7f860415ac10 2024-03-16T20:11:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041f6a90 con 0x7f860415ac10 2024-03-16T20:11:30.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.616+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860406def0 con 0x7f860415ac10 2024-03-16T20:11:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.652+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85fc00ab90 con 0x7f860415ac10 2024-03-16T20:11:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.652+0000 7f86017fa640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 0x7f85e8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.652+0000 7f8608b8f640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 0x7f85e8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.652+0000 7f8608b8f640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 0x7f85e8069350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009b60 tx=0x7f85f8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:30.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.916+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f85fc00ae40 con 0x7f860415ac10 2024-03-16T20:11:30.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:30.916+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f85fc052030 con 0x7f860415ac10 2024-03-16T20:11:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.176+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860406cab0 con 0x7f860415ac10 2024-03-16T20:11:31.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f86017fa640 1 -- 172.21.15.32:0/1176717645 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f85fc0556f0 con 0x7f860415ac10 2024-03-16T20:11:31.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:31.192 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:31.192 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:31.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 msgr2=0x7f85e8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:31.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 0x7f85e8069350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009b60 tx=0x7f85f8009190 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 msgr2=0x7f86040ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00c730 tx=0x7f85fc00cbf0 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 shutdown_connections 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f85e8066e90 0x7f85e8069350 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860415ac10 0x7f86040ccca0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86041540c0 0x7f86040cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 --2- 172.21.15.32:0/1176717645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f86040c8720 0x7f8604143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 >> 172.21.15.32:0/1176717645 conn(0x7f8604070f90 msgr2=0x7f86041591f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 shutdown_connections 2024-03-16T20:11:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:31.200+0000 7f860adf3640 1 -- 172.21.15.32:0/1176717645 wait complete. 2024-03-16T20:11:31.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:31.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:32.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 Processor -- start 2024-03-16T20:11:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 -- start start 2024-03-16T20:11:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cd4072a10 0x7f7cd4072e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd406bae0 con 0x7f7cd40c2d70 2024-03-16T20:11:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd415a540 con 0x7f7cd4072a10 2024-03-16T20:11:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.944+0000 7f7cdc711640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40c7fd0 con 0x7f7cd4154350 2024-03-16T20:11:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.948+0000 7f7cdacae640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cd4072a10 0x7f7cd4072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.948+0000 7f7cdacae640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cd4072a10 0x7f7cd4072e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48986/0 (socket says 172.21.15.32:48986) 2024-03-16T20:11:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.948+0000 7f7cdacae640 1 -- 172.21.15.32:0/2675039968 learned_addr learned my addr 172.21.15.32:0/2675039968 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.948+0000 7f7cda4ad640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.948+0000 7f7cdacae640 1 -- 172.21.15.32:0/2675039968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 msgr2=0x7f7cd406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.952+0000 7f7cdacae640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.952+0000 7f7cdacae640 1 -- 172.21.15.32:0/2675039968 >> v1:172.21.15.32:6789/0 conn(0x7f7cd40c2d70 legacy=0x7f7cd415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.952+0000 7f7cdacae640 1 -- 172.21.15.32:0/2675039968 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd406bd10 con 0x7f7cd4072a10 2024-03-16T20:11:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.956+0000 7f7cdacae640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd4072a10 0x7f7cd4072e60 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009e50 tx=0x7f7ccc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87449ff0b7b0a350 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cd94ab640 1 -- 172.21.15.32:0/2675039968 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0a5660 con 0x7f7cd4072a10 2024-03-16T20:11:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cd94ab640 1 -- 172.21.15.32:0/2675039968 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0a57c0 con 0x7f7cd4072a10 2024-03-16T20:11:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cd94ab640 1 -- 172.21.15.32:0/2675039968 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0a5b50 con 0x7f7cd4072a10 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- 172.21.15.32:0/2675039968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd4072a10 msgr2=0x7f7cd4072e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd4072a10 0x7f7cd4072e60 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009e50 tx=0x7f7ccc01c9a0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- 172.21.15.32:0/2675039968 shutdown_connections 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd4072a10 0x7f7cd4072e60 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2675039968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- 172.21.15.32:0/2675039968 >> 172.21.15.32:0/2675039968 conn(0x7f7cd406afc0 msgr2=0x7f7cd406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- 172.21.15.32:0/2675039968 shutdown_connections 2024-03-16T20:11:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- 172.21.15.32:0/2675039968 wait complete. 2024-03-16T20:11:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 Processor -- start 2024-03-16T20:11:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.964+0000 7f7cdc711640 1 -- start start 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cd4072a10 0x7f7cd415d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd41f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd41f6a70 con 0x7f7cd40c2d70 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40c79c0 con 0x7f7cd4072a10 2024-03-16T20:11:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdc711640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd41f5b80 con 0x7f7cd4154350 2024-03-16T20:11:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cd9cac640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cd9cac640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49002/0 (socket says 172.21.15.32:49002) 2024-03-16T20:11:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cd9cac640 1 -- 172.21.15.32:0/2945487381 learned_addr learned my addr 172.21.15.32:0/2945487381 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cda4ad640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cd4072a10 0x7f7cd415d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.968+0000 7f7cdacae640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd41f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 -- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 msgr2=0x7f7cd41f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd41f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 -- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cd4072a10 msgr2=0x7f7cd415d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cd4072a10 0x7f7cd415d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 -- 172.21.15.32:0/2945487381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc009b30 con 0x7f7cd40c2d70 2024-03-16T20:11:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.972+0000 7f7cd9cac640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400c420 tx=0x7f7cc400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40052d0 con 0x7f7cd40c2d70 2024-03-16T20:11:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4005430 con 0x7f7cd40c2d70 2024-03-16T20:11:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd41f5e60 con 0x7f7cd40c2d70 2024-03-16T20:11:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4007260 con 0x7f7cd40c2d70 2024-03-16T20:11:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd41f6420 con 0x7f7cd40c2d70 2024-03-16T20:11:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:32.980+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd40674e0 con 0x7f7cd40c2d70 2024-03-16T20:11:32.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.004+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7cc40073c0 con 0x7f7cd40c2d70 2024-03-16T20:11:32.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.004+0000 7f7ccaffd640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 0x7f7cb8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:32.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.004+0000 7f7cda4ad640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 0x7f7cb8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.004+0000 7f7cda4ad640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 0x7f7cb8069350 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009ac0 tx=0x7f7cd0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:33.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.340+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f7cc4007670 con 0x7f7cd40c2d70 2024-03-16T20:11:33.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.340+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f7cc4051470 con 0x7f7cd40c2d70 2024-03-16T20:11:33.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.648+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cd406e930 con 0x7f7cd40c2d70 2024-03-16T20:11:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.672+0000 7f7ccaffd640 1 -- 172.21.15.32:0/2945487381 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f7cc4020020 con 0x7f7cd40c2d70 2024-03-16T20:11:33.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:33.664 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:33.664 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 msgr2=0x7f7cb8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 0x7f7cb8069350 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009ac0 tx=0x7f7cd0009190 comp rx=0 tx=0).stop 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 msgr2=0x7f7cd4158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400c420 tx=0x7f7cc400c8e0 comp rx=0 tx=0).stop 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 shutdown_connections 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f7cb8066e90 0x7f7cb8069350 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7cd4154350 0x7f7cd41f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cd40c2d70 0x7f7cd4158190 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 --2- 172.21.15.32:0/2945487381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cd4072a10 0x7f7cd415d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 >> 172.21.15.32:0/2945487381 conn(0x7f7cd406afc0 msgr2=0x7f7cd40c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 shutdown_connections 2024-03-16T20:11:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:33.676+0000 7f7cdc711640 1 -- 172.21.15.32:0/2945487381 wait complete. 2024-03-16T20:11:34.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:34.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:35.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:11:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 Processor -- start 2024-03-16T20:11:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 -- start start 2024-03-16T20:11:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 0x7f329406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3294157eb0 con 0x7f32940cdf60 2024-03-16T20:11:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940c4c80 con 0x7f329406f8b0 2024-03-16T20:11:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f329a0f2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32940c4f40 con 0x7f32940cdb80 2024-03-16T20:11:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f3293fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f32940cdf60 0x7f3294158340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33038/0 (socket says 172.21.15.32:33038) 2024-03-16T20:11:35.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f3293fff640 1 -- 172.21.15.32:0/3814133894 learned_addr learned my addr 172.21.15.32:0/3814133894 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:35.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.408+0000 7f32937fe640 1 --2- 172.21.15.32:0/3814133894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 0x7f329406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.412+0000 7f32927fc640 1 -- 172.21.15.32:0/3814133894 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186295119 0 0) 0x7f3294157eb0 con 0x7f32940cdf60 2024-03-16T20:11:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.412+0000 7f32927fc640 1 -- 172.21.15.32:0/3814133894 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c003530 con 0x7f32940cdf60 2024-03-16T20:11:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.416+0000 7f32937fe640 1 -- 172.21.15.32:0/3814133894 >> v1:172.21.15.59:6789/0 conn(0x7f329406f8b0 legacy=0x7f3294151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.416+0000 7f32937fe640 1 -- 172.21.15.32:0/3814133894 >> v1:172.21.15.32:6789/0 conn(0x7f32940cdf60 legacy=0x7f3294158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.416+0000 7f32937fe640 1 -- 172.21.15.32:0/3814133894 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940c5220 con 0x7f32940cdb80 2024-03-16T20:11:35.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.420+0000 7f32937fe640 1 --2- 172.21.15.32:0/3814133894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 0x7f329406f370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32840097f0 tx=0x7f328401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0c74cf0f61598d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:35.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f32927fc640 1 -- 172.21.15.32:0/3814133894 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840aa430 con 0x7f32940cdb80 2024-03-16T20:11:35.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f32927fc640 1 -- 172.21.15.32:0/3814133894 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32840aa590 con 0x7f32940cdb80 2024-03-16T20:11:35.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f32927fc640 1 -- 172.21.15.32:0/3814133894 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840aa920 con 0x7f32940cdb80 2024-03-16T20:11:35.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 -- 172.21.15.32:0/3814133894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 msgr2=0x7f329406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:35.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 --2- 172.21.15.32:0/3814133894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 0x7f329406f370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32840097f0 tx=0x7f328401c4b0 comp rx=0 tx=0).stop 2024-03-16T20:11:35.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 -- 172.21.15.32:0/3814133894 shutdown_connections 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 --2- 172.21.15.32:0/3814133894 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32940cdb80 0x7f329406f370 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 -- 172.21.15.32:0/3814133894 >> 172.21.15.32:0/3814133894 conn(0x7f3294066ea0 msgr2=0x7f32940672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 -- 172.21.15.32:0/3814133894 shutdown_connections 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.428+0000 7f329a0f2640 1 -- 172.21.15.32:0/3814133894 wait complete. 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 Processor -- start 2024-03-16T20:11:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 -- start start 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32940cdf60 0x7f329414f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32941f39d0 0x7f32941f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32941f43c0 con 0x7f32940cdb80 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3294157c70 con 0x7f32940cdf60 2024-03-16T20:11:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f329a0f2640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3294157990 con 0x7f32941f39d0 2024-03-16T20:11:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f32937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f32937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49008/0 (socket says 172.21.15.32:49008) 2024-03-16T20:11:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f32937fe640 1 -- 172.21.15.32:0/772355529 learned_addr learned my addr 172.21.15.32:0/772355529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f3292ffd640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32940cdf60 0x7f329414f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:35.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.432+0000 7f3293fff640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32941f39d0 0x7f32941f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.436+0000 7f32937fe640 1 -- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32941f39d0 msgr2=0x7f32941f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.436+0000 7f32937fe640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32941f39d0 0x7f32941f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.436+0000 7f32937fe640 1 -- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32940cdf60 msgr2=0x7f329414f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.436+0000 7f32937fe640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32940cdf60 0x7f329414f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.436+0000 7f32937fe640 1 -- 172.21.15.32:0/772355529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009420 con 0x7f32940cdb80 2024-03-16T20:11:35.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.448+0000 7f32937fe640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f32840190c0 tx=0x7f328401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.452+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840aa430 con 0x7f32940cdb80 2024-03-16T20:11:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.452+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32840b4830 con 0x7f32940cdb80 2024-03-16T20:11:35.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.452+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329414d110 con 0x7f32940cdb80 2024-03-16T20:11:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.452+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840b4be0 con 0x7f32940cdb80 2024-03-16T20:11:35.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.452+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32941f34b0 con 0x7f32940cdb80 2024-03-16T20:11:35.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.456+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32941f36f0 con 0x7f32940cdb80 2024-03-16T20:11:35.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.488+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f32840b4d40 con 0x7f32940cdb80 2024-03-16T20:11:35.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.488+0000 7f327bfff640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 0x7f3280069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:35.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.488+0000 7f3292ffd640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 0x7f3280069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:35.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.488+0000 7f3292ffd640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 0x7f3280069300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f32940cd2f0 tx=0x7f3288009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:35.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.492+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f32840f03b0 con 0x7f32940cdb80 2024-03-16T20:11:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:35.748+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f32840ec650 con 0x7f32940cdb80 2024-03-16T20:11:36.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.016+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32941f36f0 con 0x7f32940cdb80 2024-03-16T20:11:36.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.040+0000 7f327bfff640 1 -- 172.21.15.32:0/772355529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5990 (secure 0 0 0) 0x7f32941f36f0 con 0x7f32940cdb80 2024-03-16T20:11:36.033 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:36.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:56:35.789410+0000","modified":"2024-03-16T20:08:00.627342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4252,"mds_2":4223,"mds_3":4243,"mds_4":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"f","rank":2,"incarnation":19,"state":"up:resolve","state_seq":30,"addr":"172.21.15.59:6833/3278441681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3278441681},{"type":"v1","addr":"172.21.15.59:6833","nonce":3278441681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4242":{"gid":4242,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":18,"addr":"172.21.15.32:6837/3949943837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3949943837},{"type":"v1","addr":"172.21.15.32:6837","nonce":3949943837}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:25.416307+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4243":{"gid":4243,"name":"a","rank":3,"incarnation":15,"state":"up:active","state_seq":17,"addr":"172.21.15.32:6835/1685908187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1685908187},{"type":"v1","addr":"172.21.15.32:6835","nonce":1685908187}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:45.556741+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.59:6837/812323509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":812323509},{"type":"v1","addr":"172.21.15.59:6837","nonce":812323509}]},"join_fscid":-1,"laggy_since":"2024-03-16T19:58:20.241138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":52,"addr":"172.21.15.32:6839/4212779160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4212779160},{"type":"v1","addr":"172.21.15.32:6839","nonce":4212779160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4252,4243,4255,4242]},"id":1}]} 2024-03-16T20:11:36.034 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-03-16T20:11:36.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 msgr2=0x7f3280069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:36.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 0x7f3280069300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f32940cd2f0 tx=0x7f3288009040 comp rx=0 tx=0).stop 2024-03-16T20:11:36.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 msgr2=0x7f3294156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f32840190c0 tx=0x7f328401acf0 comp rx=0 tx=0).stop 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 shutdown_connections 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f3280066e40 0x7f3280069300 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32941f39d0 0x7f32941f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32940cdf60 0x7f329414f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 --2- 172.21.15.32:0/772355529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32940cdb80 0x7f3294156980 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 >> 172.21.15.32:0/772355529 conn(0x7f3294066ea0 msgr2=0x7f3294158340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 shutdown_connections 2024-03-16T20:11:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:36.044+0000 7f329a0f2640 1 -- 172.21.15.32:0/772355529 wait complete. 2024-03-16T20:11:36.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:56:35.789410+0000', 'modified': '2024-03-16T20:08:00.627342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4252, 'mds_2': 4223, 'mds_3': 4243, 'mds_4': 4242}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4223': {'gid': 4223, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 30, 'addr': '172.21.15.59:6833/3278441681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 3278441681}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 3278441681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4242': {'gid': 4242, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.32:6837/3949943837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3949943837}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3949943837}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:25.416307+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4243': {'gid': 4243, 'name': 'a', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.32:6835/1685908187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1685908187}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1685908187}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:45.556741+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.59:6837/812323509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 812323509}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 812323509}]}, 'join_fscid': -1, 'laggy_since': '2024-03-16T19:58:20.241138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 52, 'addr': '172.21.15.32:6839/4212779160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4212779160}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4212779160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4252, 4243, 4255, 4242]} 2024-03-16T20:11:36.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4223:up:resolve 2024-03-16T20:11:37.658 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-16T19:56:35.789410+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4223": { "addr": "172.21.15.59:6833/3278441681", "addrs": { "addrvec": [ { "addr": "172.21.15.59:6832", "nonce": 3278441681, "type": "v2" }, { "addr": "172.21.15.59:6833", "nonce": 3278441681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4223, "incarnation": 19, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:resolve", "state_seq": 30 }, "gid_4242": { "addr": "172.21.15.32:6837/3949943837", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6836", "nonce": 3949943837, "type": "v2" }, { "addr": "172.21.15.32:6837", "nonce": 3949943837, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4242, "incarnation": 17, "join_fscid": -1, "laggy_since": "2024-03-16T19:58:25.416307+0000", "name": "e", "rank": 4, "state": "up:active", "state_seq": 18 }, "gid_4243": { "addr": "172.21.15.32:6835/1685908187", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6834", "nonce": 1685908187, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 1685908187, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4243, "incarnation": 15, "join_fscid": -1, "laggy_since": "2024-03-16T19:58:45.556741+0000", "name": "a", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_4252": { "addr": "172.21.15.59:6837/812323509", "addrs": { "addrvec": [ { "addr": "172.21.15.59:6836", "nonce": 812323509, "type": "v2" }, { "addr": "172.21.15.59:6837", "nonce": 812323509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4252, "incarnation": 10, "join_fscid": -1, "laggy_since": "2024-03-16T19:58:20.241138+0000", "name": "d", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_4255": { "addr": "172.21.15.32:6839/4212779160", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6838", "nonce": 4212779160, "type": "v2" }, { "addr": "172.21.15.32:6839", "nonce": 4212779160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4255, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 52 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-16T20:08:00.627342+0000", "qdb_cluster": [ 4252, 4243, 4255, 4242 ], "qdb_leader": 4255, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4255, "mds_1": 4252, "mds_2": 4223, "mds_3": 4243, "mds_4": 4242 } } } ], "standbys": [] } 2024-03-16T20:11:37.658 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-16T20:11:37.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:11:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 Processor -- start 2024-03-16T20:11:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 -- start start 2024-03-16T20:11:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40ac151ef0 0x7f40ac0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40ac0c97b0 0x7f40ac0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac158c80 con 0x7f40ac0c8720 2024-03-16T20:11:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac0b0ad0 con 0x7f40ac0c97b0 2024-03-16T20:11:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b42a6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac157cb0 con 0x7f40ac151ef0 2024-03-16T20:11:37.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b1841640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40ac0c97b0 0x7f40ac0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b1841640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40ac0c97b0 0x7f40ac0c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49016/0 (socket says 172.21.15.32:49016) 2024-03-16T20:11:37.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.776+0000 7f40b1841640 1 -- 172.21.15.32:0/3279735658 learned_addr learned my addr 172.21.15.32:0/3279735658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.780+0000 7f40b2042640 1 --2- 172.21.15.32:0/3279735658 >> v2:172.21.15.59:3300/0 conn(0x7f40ac151ef0 0x7f40ac0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.780+0000 7f40b1841640 1 -- 172.21.15.32:0/3279735658 >> v2:172.21.15.59:3300/0 conn(0x7f40ac151ef0 msgr2=0x7f40ac0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.780+0000 7f40b1841640 1 --2- 172.21.15.32:0/3279735658 >> v2:172.21.15.59:3300/0 conn(0x7f40ac151ef0 0x7f40ac0c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.780+0000 7f40b1841640 1 -- 172.21.15.32:0/3279735658 >> v1:172.21.15.59:6789/0 conn(0x7f40ac0c8720 legacy=0x7f40ac15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.780+0000 7f40b1841640 1 -- 172.21.15.32:0/3279735658 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac15bfc0 con 0x7f40ac0c97b0 2024-03-16T20:11:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.784+0000 7f40b1841640 1 --2- 172.21.15.32:0/3279735658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c97b0 0x7f40ac0c81e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0cb3a0 tx=0x7f409c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2756207df1868a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b1040640 1 -- 172.21.15.32:0/3279735658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0a89d0 con 0x7f40ac0c97b0 2024-03-16T20:11:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b1040640 1 -- 172.21.15.32:0/3279735658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c0a8b30 con 0x7f40ac0c97b0 2024-03-16T20:11:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b1040640 1 -- 172.21.15.32:0/3279735658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0b0440 con 0x7f40ac0c97b0 2024-03-16T20:11:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 -- 172.21.15.32:0/3279735658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c97b0 msgr2=0x7f40ac0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3279735658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c97b0 0x7f40ac0c81e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0cb3a0 tx=0x7f409c01ecd0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 -- 172.21.15.32:0/3279735658 shutdown_connections 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3279735658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c97b0 0x7f40ac0c81e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3279735658 >> v2:172.21.15.59:3300/0 conn(0x7f40ac151ef0 0x7f40ac0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 -- 172.21.15.32:0/3279735658 >> 172.21.15.32:0/3279735658 conn(0x7f40ac070f90 msgr2=0x7f40ac0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 -- 172.21.15.32:0/3279735658 shutdown_connections 2024-03-16T20:11:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.788+0000 7f40b42a6640 1 -- 172.21.15.32:0/3279735658 wait complete. 2024-03-16T20:11:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 Processor -- start 2024-03-16T20:11:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 -- start start 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40ac0c97b0 0x7f40ac1f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40ac151ef0 0x7f40ac0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1f1060 con 0x7f40ac0c8720 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1f1620 con 0x7f40ac0c97b0 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b42a6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac0c5030 con 0x7f40ac151ef0 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b2042640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b2042640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49026/0 (socket says 172.21.15.32:49026) 2024-03-16T20:11:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b2042640 1 -- 172.21.15.32:0/3457246553 learned_addr learned my addr 172.21.15.32:0/3457246553 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.792+0000 7f40b1841640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40ac0c97b0 0x7f40ac1f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2843640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40ac151ef0 0x7f40ac0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2042640 1 -- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40ac151ef0 msgr2=0x7f40ac0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2042640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40ac151ef0 0x7f40ac0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2042640 1 -- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40ac0c97b0 msgr2=0x7f40ac1f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2042640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40ac0c97b0 0x7f40ac1f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.796+0000 7f40b2042640 1 -- 172.21.15.32:0/3457246553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0094d0 con 0x7f40ac0c8720 2024-03-16T20:11:37.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.800+0000 7f40b2042640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f40a800c7f0 tx=0x7f40a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a8007820 con 0x7f40ac0c8720 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a8007980 con 0x7f40ac0c8720 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac0c52f0 con 0x7f40ac0c8720 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a8007d30 con 0x7f40ac0c8720 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac1f12b0 con 0x7f40ac0c8720 2024-03-16T20:11:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.804+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac068000 con 0x7f40ac0c8720 2024-03-16T20:11:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.840+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f40a8024480 con 0x7f40ac0c8720 2024-03-16T20:11:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.840+0000 7f40a2ffd640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 0x7f4090069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.840+0000 7f40b1841640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 0x7f4090069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:37.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:37.844+0000 7f40b1841640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 0x7f4090069350 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b2000 tx=0x7f409c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:38.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.132+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f40a80563b0 con 0x7f40ac0c8720 2024-03-16T20:11:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.132+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f40a8051f50 con 0x7f40ac0c8720 2024-03-16T20:11:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.292+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40ac158480 con 0x7f4090066e90 2024-03-16T20:11:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.300+0000 7f40a2ffd640 1 -- 172.21.15.32:0/3457246553 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332622 (secure 0 0 0) 0x7f409c0a87d0 con 0x7f4090066e90 2024-03-16T20:11:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:38.299 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:11:38.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 msgr2=0x7f4090069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:38.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 0x7f4090069350 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b2000 tx=0x7f409c008040 comp rx=0 tx=0).stop 2024-03-16T20:11:38.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 msgr2=0x7f40ac1edaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:38.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f40a800c7f0 tx=0x7f40a800ccb0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 shutdown_connections 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4090066e90 0x7f4090069350 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f40ac151ef0 0x7f40ac0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40ac0c97b0 0x7f40ac1f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 --2- 172.21.15.32:0/3457246553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ac0c8720 0x7f40ac1edaf0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 >> 172.21.15.32:0/3457246553 conn(0x7f40ac070f90 msgr2=0x7f40ac06efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 shutdown_connections 2024-03-16T20:11:38.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.304+0000 7f40b42a6640 1 -- 172.21.15.32:0/3457246553 wait complete. 2024-03-16T20:11:38.314 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-03-16T20:11:36.845781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236864,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737856,"statfs":{"total":773094113280,"available":772851564544,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003791"},"pg_stats":[{"pgid":"2.1f","version":"32'14","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.407709+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T20:09:03.407709+0000","last_peered":"2024-03-16T20:09:03.407709+0000","last_clean":"2024-03-16T20:09:03.407709+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:09:03.407709+0000","last_undegraded":"2024-03-16T20:09:03.407709+0000","last_fullsized":"2024-03-16T20:09:03.407709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.473025+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T20:08:38.473025+0000","last_peered":"2024-03-16T20:08:38.473025+0000","last_clean":"2024-03-16T20:08:38.473025+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:08:38.473025+0000","last_undegraded":"2024-03-16T20:08:38.473025+0000","last_fullsized":"2024-03-16T20:08:38.473025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474069+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T20:08:48.474069+0000","last_peered":"2024-03-16T20:08:48.474069+0000","last_clean":"2024-03-16T20:08:48.474069+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:08:48.474069+0000","last_undegraded":"2024-03-16T20:08:48.474069+0000","last_fullsized":"2024-03-16T20:08:48.474069+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.472417+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T20:08:33.472417+0000","last_peered":"2024-03-16T20:08:33.472417+0000","last_clean":"2024-03-16T20:08:33.472417+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:08:33.472417+0000","last_undegraded":"2024-03-16T20:08:33.472417+0000","last_fullsized":"2024-03-16T20:08:33.472417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408303+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T20:09:08.408303+0000","last_peered":"2024-03-16T20:09:08.408303+0000","last_clean":"2024-03-16T20:09:08.408303+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:09:08.408303+0000","last_undegraded":"2024-03-16T20:09:08.408303+0000","last_fullsized":"2024-03-16T20:09:08.408303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.479106+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T20:08:49.479106+0000","last_peered":"2024-03-16T20:08:49.479106+0000","last_clean":"2024-03-16T20:08:49.479106+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:08:49.479106+0000","last_undegraded":"2024-03-16T20:08:49.479106+0000","last_fullsized":"2024-03-16T20:08:49.479106+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'13","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405198+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T20:09:04.405198+0000","last_peered":"2024-03-16T20:09:04.405198+0000","last_clean":"2024-03-16T20:09:04.405198+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:09:04.405198+0000","last_undegraded":"2024-03-16T20:09:04.405198+0000","last_fullsized":"2024-03-16T20:09:04.405198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411271+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T20:06:39.411271+0000","last_peered":"2024-03-16T20:06:39.411271+0000","last_clean":"2024-03-16T20:06:39.411271+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:06:39.411271+0000","last_undegraded":"2024-03-16T20:06:39.411271+0000","last_fullsized":"2024-03-16T20:06:39.411271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464297+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T20:06:44.464297+0000","last_peered":"2024-03-16T20:06:44.464297+0000","last_clean":"2024-03-16T20:06:44.464297+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:06:44.464297+0000","last_undegraded":"2024-03-16T20:06:44.464297+0000","last_fullsized":"2024-03-16T20:06:44.464297+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405162+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T20:09:04.405162+0000","last_peered":"2024-03-16T20:09:04.405162+0000","last_clean":"2024-03-16T20:09:04.405162+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:09:04.405162+0000","last_undegraded":"2024-03-16T20:09:04.405162+0000","last_fullsized":"2024-03-16T20:09:04.405162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411290+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T20:06:39.411290+0000","last_peered":"2024-03-16T20:06:39.411290+0000","last_clean":"2024-03-16T20:06:39.411290+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:06:39.411290+0000","last_undegraded":"2024-03-16T20:06:39.411290+0000","last_fullsized":"2024-03-16T20:06:39.411290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464225+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T20:06:44.464225+0000","last_peered":"2024-03-16T20:06:44.464225+0000","last_clean":"2024-03-16T20:06:44.464225+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:06:44.464225+0000","last_undegraded":"2024-03-16T20:06:44.464225+0000","last_fullsized":"2024-03-16T20:06:44.464225+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448622+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T20:07:29.448622+0000","last_peered":"2024-03-16T20:07:29.448622+0000","last_clean":"2024-03-16T20:07:29.448622+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:07:29.448622+0000","last_undegraded":"2024-03-16T20:07:29.448622+0000","last_fullsized":"2024-03-16T20:07:29.448622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.395875+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T20:07:49.395875+0000","last_peered":"2024-03-16T20:07:49.395875+0000","last_clean":"2024-03-16T20:07:49.395875+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:07:49.395875+0000","last_undegraded":"2024-03-16T20:07:49.395875+0000","last_fullsized":"2024-03-16T20:07:49.395875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473222+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T20:07:59.473222+0000","last_peered":"2024-03-16T20:07:59.473222+0000","last_clean":"2024-03-16T20:07:59.473222+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:07:59.473222+0000","last_undegraded":"2024-03-16T20:07:59.473222+0000","last_fullsized":"2024-03-16T20:07:59.473222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467453+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T20:07:53.467453+0000","last_peered":"2024-03-16T20:07:53.467453+0000","last_clean":"2024-03-16T20:07:53.467453+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:07:53.467453+0000","last_undegraded":"2024-03-16T20:07:53.467453+0000","last_fullsized":"2024-03-16T20:07:53.467453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635909+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T20:07:25.635909+0000","last_peered":"2024-03-16T20:07:25.635909+0000","last_clean":"2024-03-16T20:07:25.635909+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:07:25.635909+0000","last_undegraded":"2024-03-16T20:07:25.635909+0000","last_fullsized":"2024-03-16T20:07:25.635909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473182+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T20:07:59.473182+0000","last_peered":"2024-03-16T20:07:59.473182+0000","last_clean":"2024-03-16T20:07:59.473182+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:07:59.473182+0000","last_undegraded":"2024-03-16T20:07:59.473182+0000","last_fullsized":"2024-03-16T20:07:59.473182+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463912+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T20:07:28.463912+0000","last_peered":"2024-03-16T20:07:28.463912+0000","last_clean":"2024-03-16T20:07:28.463912+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:07:28.463912+0000","last_undegraded":"2024-03-16T20:07:28.463912+0000","last_fullsized":"2024-03-16T20:07:28.463912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635877+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T20:07:25.635877+0000","last_peered":"2024-03-16T20:07:25.635877+0000","last_clean":"2024-03-16T20:07:25.635877+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:07:25.635877+0000","last_undegraded":"2024-03-16T20:07:25.635877+0000","last_fullsized":"2024-03-16T20:07:25.635877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:00.639712+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T20:08:00.639712+0000","last_peered":"2024-03-16T20:08:00.639712+0000","last_clean":"2024-03-16T20:08:00.639712+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:08:00.639712+0000","last_undegraded":"2024-03-16T20:08:00.639712+0000","last_fullsized":"2024-03-16T20:08:00.639712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.469295+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T20:07:29.469295+0000","last_peered":"2024-03-16T20:07:29.469295+0000","last_clean":"2024-03-16T20:07:29.469295+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:07:29.469295+0000","last_undegraded":"2024-03-16T20:07:29.469295+0000","last_fullsized":"2024-03-16T20:07:29.469295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.392918+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T20:07:24.392918+0000","last_peered":"2024-03-16T20:07:24.392918+0000","last_clean":"2024-03-16T20:07:24.392918+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:07:24.392918+0000","last_undegraded":"2024-03-16T20:07:24.392918+0000","last_fullsized":"2024-03-16T20:07:24.392918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473238+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T20:07:59.473238+0000","last_peered":"2024-03-16T20:07:59.473238+0000","last_clean":"2024-03-16T20:07:59.473238+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:07:59.473238+0000","last_undegraded":"2024-03-16T20:07:59.473238+0000","last_fullsized":"2024-03-16T20:07:59.473238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.395991+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T20:07:28.395991+0000","last_peered":"2024-03-16T20:07:28.395991+0000","last_clean":"2024-03-16T20:07:28.395991+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:07:28.395991+0000","last_undegraded":"2024-03-16T20:07:28.395991+0000","last_fullsized":"2024-03-16T20:07:28.395991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.448140+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T20:07:24.448140+0000","last_peered":"2024-03-16T20:07:24.448140+0000","last_clean":"2024-03-16T20:07:24.448140+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:07:24.448140+0000","last_undegraded":"2024-03-16T20:07:24.448140+0000","last_fullsized":"2024-03-16T20:07:24.448140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.481902+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T20:09:14.481902+0000","last_peered":"2024-03-16T20:09:14.481902+0000","last_clean":"2024-03-16T20:09:14.481902+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:09:14.481902+0000","last_undegraded":"2024-03-16T20:09:14.481902+0000","last_fullsized":"2024-03-16T20:09:14.481902+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.417264+0000","last_change":"2024-03-16T19:57:41.287408+0000","last_active":"2024-03-16T20:08:33.417264+0000","last_peered":"2024-03-16T20:08:33.417264+0000","last_clean":"2024-03-16T20:08:33.417264+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:08:33.417264+0000","last_undegraded":"2024-03-16T20:08:33.417264+0000","last_fullsized":"2024-03-16T20:08:33.417264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:54.981214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635961+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T20:07:25.635961+0000","last_peered":"2024-03-16T20:07:25.635961+0000","last_clean":"2024-03-16T20:07:25.635961+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:07:25.635961+0000","last_undegraded":"2024-03-16T20:07:25.635961+0000","last_fullsized":"2024-03-16T20:07:25.635961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473514+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T20:08:43.473514+0000","last_peered":"2024-03-16T20:08:43.473514+0000","last_clean":"2024-03-16T20:08:43.473514+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:08:43.473514+0000","last_undegraded":"2024-03-16T20:08:43.473514+0000","last_fullsized":"2024-03-16T20:08:43.473514+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647671+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T20:09:05.647671+0000","last_peered":"2024-03-16T20:09:05.647671+0000","last_clean":"2024-03-16T20:09:05.647671+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:09:05.647671+0000","last_undegraded":"2024-03-16T20:09:05.647671+0000","last_fullsized":"2024-03-16T20:09:05.647671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405217+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T20:09:04.405217+0000","last_peered":"2024-03-16T20:09:04.405217+0000","last_clean":"2024-03-16T20:09:04.405217+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:09:04.405217+0000","last_undegraded":"2024-03-16T20:09:04.405217+0000","last_fullsized":"2024-03-16T20:09:04.405217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:13.408811+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T20:09:13.408811+0000","last_peered":"2024-03-16T20:09:13.408811+0000","last_clean":"2024-03-16T20:09:13.408811+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:09:13.408811+0000","last_undegraded":"2024-03-16T20:09:13.408811+0000","last_fullsized":"2024-03-16T20:09:13.408811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432177+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T20:09:09.432177+0000","last_peered":"2024-03-16T20:09:09.432177+0000","last_clean":"2024-03-16T20:09:09.432177+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:09:09.432177+0000","last_undegraded":"2024-03-16T20:09:09.432177+0000","last_fullsized":"2024-03-16T20:09:09.432177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647700+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T20:09:05.647700+0000","last_peered":"2024-03-16T20:09:05.647700+0000","last_clean":"2024-03-16T20:09:05.647700+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:09:05.647700+0000","last_undegraded":"2024-03-16T20:09:05.647700+0000","last_fullsized":"2024-03-16T20:09:05.647700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648736+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T20:09:15.648736+0000","last_peered":"2024-03-16T20:09:15.648736+0000","last_clean":"2024-03-16T20:09:15.648736+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:09:15.648736+0000","last_undegraded":"2024-03-16T20:09:15.648736+0000","last_fullsized":"2024-03-16T20:09:15.648736+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.481151+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T20:09:09.481151+0000","last_peered":"2024-03-16T20:09:09.481151+0000","last_clean":"2024-03-16T20:09:09.481151+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:09:09.481151+0000","last_undegraded":"2024-03-16T20:09:09.481151+0000","last_fullsized":"2024-03-16T20:09:09.481151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'6","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431117+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T20:09:04.431117+0000","last_peered":"2024-03-16T20:09:04.431117+0000","last_clean":"2024-03-16T20:09:04.431117+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:09:04.431117+0000","last_undegraded":"2024-03-16T20:09:04.431117+0000","last_fullsized":"2024-03-16T20:09:04.431117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648771+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T20:09:15.648771+0000","last_peered":"2024-03-16T20:09:15.648771+0000","last_clean":"2024-03-16T20:09:15.648771+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:09:15.648771+0000","last_undegraded":"2024-03-16T20:09:15.648771+0000","last_fullsized":"2024-03-16T20:09:15.648771+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:35.643746+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T20:08:35.643746+0000","last_peered":"2024-03-16T20:08:35.643746+0000","last_clean":"2024-03-16T20:08:35.643746+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:08:35.643746+0000","last_undegraded":"2024-03-16T20:08:35.643746+0000","last_fullsized":"2024-03-16T20:08:35.643746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.420300+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T20:09:03.420300+0000","last_peered":"2024-03-16T20:09:03.420300+0000","last_clean":"2024-03-16T20:09:03.420300+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:09:03.420300+0000","last_undegraded":"2024-03-16T20:09:03.420300+0000","last_fullsized":"2024-03-16T20:09:03.420300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:35.636979+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T20:07:35.636979+0000","last_peered":"2024-03-16T20:07:35.636979+0000","last_clean":"2024-03-16T20:07:35.636979+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:07:35.636979+0000","last_undegraded":"2024-03-16T20:07:35.636979+0000","last_fullsized":"2024-03-16T20:07:35.636979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408207+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T20:09:08.408207+0000","last_peered":"2024-03-16T20:09:08.408207+0000","last_clean":"2024-03-16T20:09:08.408207+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:09:08.408207+0000","last_undegraded":"2024-03-16T20:09:08.408207+0000","last_fullsized":"2024-03-16T20:09:08.408207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:34.401181+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T20:08:34.401181+0000","last_peered":"2024-03-16T20:08:34.401181+0000","last_clean":"2024-03-16T20:08:34.401181+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:08:34.401181+0000","last_undegraded":"2024-03-16T20:08:34.401181+0000","last_fullsized":"2024-03-16T20:08:34.401181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:33.396597+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T20:07:33.396597+0000","last_peered":"2024-03-16T20:07:33.396597+0000","last_clean":"2024-03-16T20:07:33.396597+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:07:33.396597+0000","last_undegraded":"2024-03-16T20:07:33.396597+0000","last_fullsized":"2024-03-16T20:07:33.396597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.476309+0000","last_change":"2024-03-16T19:57:41.238831+0000","last_active":"2024-03-16T20:09:08.476309+0000","last_peered":"2024-03-16T20:09:08.476309+0000","last_clean":"2024-03-16T20:09:08.476309+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:09:08.476309+0000","last_undegraded":"2024-03-16T20:09:08.476309+0000","last_fullsized":"2024-03-16T20:09:08.476309+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:56:28.075437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431163+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T20:09:04.431163+0000","last_peered":"2024-03-16T20:09:04.431163+0000","last_clean":"2024-03-16T20:09:04.431163+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:09:04.431163+0000","last_undegraded":"2024-03-16T20:09:04.431163+0000","last_fullsized":"2024-03-16T20:09:04.431163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":197,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396571+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T20:07:54.396571+0000","last_peered":"2024-03-16T20:07:54.396571+0000","last_clean":"2024-03-16T20:07:54.396571+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:07:54.396571+0000","last_undegraded":"2024-03-16T20:07:54.396571+0000","last_fullsized":"2024-03-16T20:07:54.396571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461272+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T20:09:14.461272+0000","last_peered":"2024-03-16T20:09:14.461272+0000","last_clean":"2024-03-16T20:09:14.461272+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:09:14.461272+0000","last_undegraded":"2024-03-16T20:09:14.461272+0000","last_fullsized":"2024-03-16T20:09:14.461272+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396635+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T20:07:54.396635+0000","last_peered":"2024-03-16T20:07:54.396635+0000","last_clean":"2024-03-16T20:07:54.396635+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:07:54.396635+0000","last_undegraded":"2024-03-16T20:07:54.396635+0000","last_fullsized":"2024-03-16T20:07:54.396635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412513+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T20:07:48.412513+0000","last_peered":"2024-03-16T20:07:48.412513+0000","last_clean":"2024-03-16T20:07:48.412513+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:07:48.412513+0000","last_undegraded":"2024-03-16T20:07:48.412513+0000","last_fullsized":"2024-03-16T20:07:48.412513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.449341+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T20:07:34.449341+0000","last_peered":"2024-03-16T20:07:34.449341+0000","last_clean":"2024-03-16T20:07:34.449341+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:07:34.449341+0000","last_undegraded":"2024-03-16T20:07:34.449341+0000","last_fullsized":"2024-03-16T20:07:34.449341+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421860+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T20:07:54.421860+0000","last_peered":"2024-03-16T20:07:54.421860+0000","last_clean":"2024-03-16T20:07:54.421860+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:07:54.421860+0000","last_undegraded":"2024-03-16T20:07:54.421860+0000","last_fullsized":"2024-03-16T20:07:54.421860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:50.638775+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T20:07:50.638775+0000","last_peered":"2024-03-16T20:07:50.638775+0000","last_clean":"2024-03-16T20:07:50.638775+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:07:50.638775+0000","last_undegraded":"2024-03-16T20:07:50.638775+0000","last_fullsized":"2024-03-16T20:07:50.638775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:58.399907+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T20:07:58.399907+0000","last_peered":"2024-03-16T20:07:58.399907+0000","last_clean":"2024-03-16T20:07:58.399907+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:07:58.399907+0000","last_undegraded":"2024-03-16T20:07:58.399907+0000","last_fullsized":"2024-03-16T20:07:58.399907+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396733+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T20:07:54.396733+0000","last_peered":"2024-03-16T20:07:54.396733+0000","last_clean":"2024-03-16T20:07:54.396733+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:07:54.396733+0000","last_undegraded":"2024-03-16T20:07:54.396733+0000","last_fullsized":"2024-03-16T20:07:54.396733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412548+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T20:07:48.412548+0000","last_peered":"2024-03-16T20:07:48.412548+0000","last_clean":"2024-03-16T20:07:48.412548+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:07:48.412548+0000","last_undegraded":"2024-03-16T20:07:48.412548+0000","last_fullsized":"2024-03-16T20:07:48.412548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.419200+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T20:07:34.419200+0000","last_peered":"2024-03-16T20:07:34.419200+0000","last_clean":"2024-03-16T20:07:34.419200+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:07:34.419200+0000","last_undegraded":"2024-03-16T20:07:34.419200+0000","last_fullsized":"2024-03-16T20:07:34.419200+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396681+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T20:07:54.396681+0000","last_peered":"2024-03-16T20:07:54.396681+0000","last_clean":"2024-03-16T20:07:54.396681+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:07:54.396681+0000","last_undegraded":"2024-03-16T20:07:54.396681+0000","last_fullsized":"2024-03-16T20:07:54.396681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.471915+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T20:07:49.471915+0000","last_peered":"2024-03-16T20:07:49.471915+0000","last_clean":"2024-03-16T20:07:49.471915+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:07:49.471915+0000","last_undegraded":"2024-03-16T20:07:49.471915+0000","last_fullsized":"2024-03-16T20:07:49.471915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.469980+0000","last_change":"2024-03-16T19:57:41.242958+0000","last_active":"2024-03-16T20:07:34.469980+0000","last_peered":"2024-03-16T20:07:34.469980+0000","last_clean":"2024-03-16T20:07:34.469980+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:07:34.469980+0000","last_undegraded":"2024-03-16T20:07:34.469980+0000","last_fullsized":"2024-03-16T20:07:34.469980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:17:20.536712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399350+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T20:08:19.399350+0000","last_peered":"2024-03-16T20:08:19.399350+0000","last_clean":"2024-03-16T20:08:19.399350+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:08:19.399350+0000","last_undegraded":"2024-03-16T20:08:19.399350+0000","last_fullsized":"2024-03-16T20:08:19.399350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'18","reported_seq":60,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.415178+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T20:08:13.415178+0000","last_peered":"2024-03-16T20:08:13.415178+0000","last_clean":"2024-03-16T20:08:13.415178+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:08:13.415178+0000","last_undegraded":"2024-03-16T20:08:13.415178+0000","last_fullsized":"2024-03-16T20:08:13.415178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.406062+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T20:08:48.406062+0000","last_peered":"2024-03-16T20:08:48.406062+0000","last_clean":"2024-03-16T20:08:48.406062+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:08:48.406062+0000","last_undegraded":"2024-03-16T20:08:48.406062+0000","last_fullsized":"2024-03-16T20:08:48.406062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.455260+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T20:08:19.455260+0000","last_peered":"2024-03-16T20:08:19.455260+0000","last_clean":"2024-03-16T20:08:19.455260+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:08:19.455260+0000","last_undegraded":"2024-03-16T20:08:19.455260+0000","last_fullsized":"2024-03-16T20:08:19.455260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'6","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454559+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T20:08:14.454559+0000","last_peered":"2024-03-16T20:08:14.454559+0000","last_clean":"2024-03-16T20:08:14.454559+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:08:14.454559+0000","last_undegraded":"2024-03-16T20:08:14.454559+0000","last_fullsized":"2024-03-16T20:08:14.454559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474130+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T20:08:48.474130+0000","last_peered":"2024-03-16T20:08:48.474130+0000","last_clean":"2024-03-16T20:08:48.474130+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:08:48.474130+0000","last_undegraded":"2024-03-16T20:08:48.474130+0000","last_fullsized":"2024-03-16T20:08:48.474130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399380+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T20:08:19.399380+0000","last_peered":"2024-03-16T20:08:19.399380+0000","last_clean":"2024-03-16T20:08:19.399380+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:08:19.399380+0000","last_undegraded":"2024-03-16T20:08:19.399380+0000","last_fullsized":"2024-03-16T20:08:19.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.424455+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T20:08:14.424455+0000","last_peered":"2024-03-16T20:08:14.424455+0000","last_clean":"2024-03-16T20:08:14.424455+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:08:14.424455+0000","last_undegraded":"2024-03-16T20:08:14.424455+0000","last_fullsized":"2024-03-16T20:08:14.424455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470857+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T20:08:23.470857+0000","last_peered":"2024-03-16T20:08:23.470857+0000","last_clean":"2024-03-16T20:08:23.470857+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:08:23.470857+0000","last_undegraded":"2024-03-16T20:08:23.470857+0000","last_fullsized":"2024-03-16T20:08:23.470857+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:03.407915+0000","last_change":"2024-03-16T19:57:41.287279+0000","last_active":"2024-03-16T20:07:03.407915+0000","last_peered":"2024-03-16T20:07:03.407915+0000","last_clean":"2024-03-16T20:07:03.407915+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:07:03.407915+0000","last_undegraded":"2024-03-16T20:07:03.407915+0000","last_fullsized":"2024-03-16T20:07:03.407915+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:46.850016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.401433+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T20:08:13.401433+0000","last_peered":"2024-03-16T20:08:13.401433+0000","last_clean":"2024-03-16T20:08:13.401433+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:08:13.401433+0000","last_undegraded":"2024-03-16T20:08:13.401433+0000","last_fullsized":"2024-03-16T20:08:13.401433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.429172+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T20:08:49.429172+0000","last_peered":"2024-03-16T20:08:49.429172+0000","last_clean":"2024-03-16T20:08:49.429172+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:08:49.429172+0000","last_undegraded":"2024-03-16T20:08:49.429172+0000","last_fullsized":"2024-03-16T20:08:49.429172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402419+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T20:08:44.402419+0000","last_peered":"2024-03-16T20:08:44.402419+0000","last_clean":"2024-03-16T20:08:44.402419+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:08:44.402419+0000","last_undegraded":"2024-03-16T20:08:44.402419+0000","last_fullsized":"2024-03-16T20:08:44.402419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454598+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T20:08:14.454598+0000","last_peered":"2024-03-16T20:08:14.454598+0000","last_clean":"2024-03-16T20:08:14.454598+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:08:14.454598+0000","last_undegraded":"2024-03-16T20:08:14.454598+0000","last_fullsized":"2024-03-16T20:08:14.454598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:25.642519+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T20:08:25.642519+0000","last_peered":"2024-03-16T20:08:25.642519+0000","last_clean":"2024-03-16T20:08:25.642519+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:08:25.642519+0000","last_undegraded":"2024-03-16T20:08:25.642519+0000","last_fullsized":"2024-03-16T20:08:25.642519+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.475987+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T20:08:19.475987+0000","last_peered":"2024-03-16T20:08:19.475987+0000","last_clean":"2024-03-16T20:08:19.475987+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:08:19.475987+0000","last_undegraded":"2024-03-16T20:08:19.475987+0000","last_fullsized":"2024-03-16T20:08:19.475987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644613+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T20:08:40.644613+0000","last_peered":"2024-03-16T20:08:40.644613+0000","last_clean":"2024-03-16T20:08:40.644613+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:08:40.644613+0000","last_undegraded":"2024-03-16T20:08:40.644613+0000","last_fullsized":"2024-03-16T20:08:40.644613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:24.476654+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T20:08:24.476654+0000","last_peered":"2024-03-16T20:08:24.476654+0000","last_clean":"2024-03-16T20:08:24.476654+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:08:24.476654+0000","last_undegraded":"2024-03-16T20:08:24.476654+0000","last_fullsized":"2024-03-16T20:08:24.476654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405275+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T20:08:43.405275+0000","last_peered":"2024-03-16T20:08:43.405275+0000","last_clean":"2024-03-16T20:08:43.405275+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:08:43.405275+0000","last_undegraded":"2024-03-16T20:08:43.405275+0000","last_fullsized":"2024-03-16T20:08:43.405275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427645+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T20:08:39.427645+0000","last_peered":"2024-03-16T20:08:39.427645+0000","last_clean":"2024-03-16T20:08:39.427645+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:08:39.427645+0000","last_undegraded":"2024-03-16T20:08:39.427645+0000","last_fullsized":"2024-03-16T20:08:39.427645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470727+0000","last_change":"2024-03-16T19:57:41.246101+0000","last_active":"2024-03-16T20:08:23.470727+0000","last_peered":"2024-03-16T20:08:23.470727+0000","last_clean":"2024-03-16T20:08:23.470727+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:08:23.470727+0000","last_undegraded":"2024-03-16T20:08:23.470727+0000","last_fullsized":"2024-03-16T20:08:23.470727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:04.608965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.428389+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T20:08:44.428389+0000","last_peered":"2024-03-16T20:08:44.428389+0000","last_clean":"2024-03-16T20:08:44.428389+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:08:44.428389+0000","last_undegraded":"2024-03-16T20:08:44.428389+0000","last_fullsized":"2024-03-16T20:08:44.428389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.457446+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T20:08:39.457446+0000","last_peered":"2024-03-16T20:08:39.457446+0000","last_clean":"2024-03-16T20:08:39.457446+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:08:39.457446+0000","last_undegraded":"2024-03-16T20:08:39.457446+0000","last_fullsized":"2024-03-16T20:08:39.457446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470791+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T20:08:23.470791+0000","last_peered":"2024-03-16T20:08:23.470791+0000","last_clean":"2024-03-16T20:08:23.470791+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:08:23.470791+0000","last_undegraded":"2024-03-16T20:08:23.470791+0000","last_fullsized":"2024-03-16T20:08:23.470791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405199+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T20:08:43.405199+0000","last_peered":"2024-03-16T20:08:43.405199+0000","last_clean":"2024-03-16T20:08:43.405199+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:08:43.405199+0000","last_undegraded":"2024-03-16T20:08:43.405199+0000","last_fullsized":"2024-03-16T20:08:43.405199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644545+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T20:08:40.644545+0000","last_peered":"2024-03-16T20:08:40.644545+0000","last_clean":"2024-03-16T20:08:40.644545+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:08:40.644545+0000","last_undegraded":"2024-03-16T20:08:40.644545+0000","last_fullsized":"2024-03-16T20:08:40.644545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427570+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T20:08:39.427570+0000","last_peered":"2024-03-16T20:08:39.427570+0000","last_clean":"2024-03-16T20:08:39.427570+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:08:39.427570+0000","last_undegraded":"2024-03-16T20:08:39.427570+0000","last_fullsized":"2024-03-16T20:08:39.427570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461319+0000","last_change":"2024-03-16T19:57:41.279710+0000","last_active":"2024-03-16T20:09:14.461319+0000","last_peered":"2024-03-16T20:09:14.461319+0000","last_clean":"2024-03-16T20:09:14.461319+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:09:14.461319+0000","last_undegraded":"2024-03-16T20:09:14.461319+0000","last_fullsized":"2024-03-16T20:09:14.461319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:02:54.725387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473548+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T20:08:43.473548+0000","last_peered":"2024-03-16T20:08:43.473548+0000","last_clean":"2024-03-16T20:08:43.473548+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:08:43.473548+0000","last_undegraded":"2024-03-16T20:08:43.473548+0000","last_fullsized":"2024-03-16T20:08:43.473548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.472974+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T20:08:38.472974+0000","last_peered":"2024-03-16T20:08:38.472974+0000","last_clean":"2024-03-16T20:08:38.472974+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:08:38.472974+0000","last_undegraded":"2024-03-16T20:08:38.472974+0000","last_fullsized":"2024-03-16T20:08:38.472974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.459824+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T20:09:04.459824+0000","last_peered":"2024-03-16T20:09:04.459824+0000","last_clean":"2024-03-16T20:09:04.459824+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:09:04.459824+0000","last_undegraded":"2024-03-16T20:09:04.459824+0000","last_fullsized":"2024-03-16T20:09:04.459824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:38.390198+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T20:06:38.390198+0000","last_peered":"2024-03-16T20:06:38.390198+0000","last_clean":"2024-03-16T20:06:38.390198+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:06:38.390198+0000","last_undegraded":"2024-03-16T20:06:38.390198+0000","last_fullsized":"2024-03-16T20:06:38.390198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.405682+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T20:06:43.405682+0000","last_peered":"2024-03-16T20:06:43.405682+0000","last_clean":"2024-03-16T20:06:43.405682+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:06:43.405682+0000","last_undegraded":"2024-03-16T20:06:43.405682+0000","last_fullsized":"2024-03-16T20:06:43.405682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.475556+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T20:09:03.475556+0000","last_peered":"2024-03-16T20:09:03.475556+0000","last_clean":"2024-03-16T20:09:03.475556+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:09:03.475556+0000","last_undegraded":"2024-03-16T20:09:03.475556+0000","last_fullsized":"2024-03-16T20:09:03.475556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.463549+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T20:06:39.463549+0000","last_peered":"2024-03-16T20:06:39.463549+0000","last_clean":"2024-03-16T20:06:39.463549+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:06:39.463549+0000","last_undegraded":"2024-03-16T20:06:39.463549+0000","last_fullsized":"2024-03-16T20:06:39.463549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.458436+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T20:06:43.458436+0000","last_peered":"2024-03-16T20:06:43.458436+0000","last_clean":"2024-03-16T20:06:43.458436+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:06:43.458436+0000","last_undegraded":"2024-03-16T20:06:43.458436+0000","last_fullsized":"2024-03-16T20:06:43.458436+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.460475+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T20:09:09.460475+0000","last_peered":"2024-03-16T20:09:09.460475+0000","last_clean":"2024-03-16T20:09:09.460475+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:09:09.460475+0000","last_undegraded":"2024-03-16T20:09:09.460475+0000","last_fullsized":"2024-03-16T20:09:09.460475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.420799+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T20:09:08.420799+0000","last_peered":"2024-03-16T20:09:08.420799+0000","last_clean":"2024-03-16T20:09:08.420799+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:09:08.420799+0000","last_undegraded":"2024-03-16T20:09:08.420799+0000","last_fullsized":"2024-03-16T20:09:08.420799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.431997+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T20:09:09.431997+0000","last_peered":"2024-03-16T20:09:09.431997+0000","last_clean":"2024-03-16T20:09:09.431997+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:09:09.431997+0000","last_undegraded":"2024-03-16T20:09:09.431997+0000","last_fullsized":"2024-03-16T20:09:09.431997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408250+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T20:09:08.408250+0000","last_peered":"2024-03-16T20:09:08.408250+0000","last_clean":"2024-03-16T20:09:08.408250+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:09:08.408250+0000","last_undegraded":"2024-03-16T20:09:08.408250+0000","last_fullsized":"2024-03-16T20:09:08.408250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463977+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T20:07:28.463977+0000","last_peered":"2024-03-16T20:07:28.463977+0000","last_clean":"2024-03-16T20:07:28.463977+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:07:28.463977+0000","last_undegraded":"2024-03-16T20:07:28.463977+0000","last_fullsized":"2024-03-16T20:07:28.463977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635930+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T20:07:25.635930+0000","last_peered":"2024-03-16T20:07:25.635930+0000","last_clean":"2024-03-16T20:07:25.635930+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:07:25.635930+0000","last_undegraded":"2024-03-16T20:07:25.635930+0000","last_fullsized":"2024-03-16T20:07:25.635930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448666+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T20:07:29.448666+0000","last_peered":"2024-03-16T20:07:29.448666+0000","last_clean":"2024-03-16T20:07:29.448666+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:07:29.448666+0000","last_undegraded":"2024-03-16T20:07:29.448666+0000","last_fullsized":"2024-03-16T20:07:29.448666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.464010+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T20:07:28.464010+0000","last_peered":"2024-03-16T20:07:28.464010+0000","last_clean":"2024-03-16T20:07:28.464010+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:07:28.464010+0000","last_undegraded":"2024-03-16T20:07:28.464010+0000","last_fullsized":"2024-03-16T20:07:28.464010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.418412+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T20:07:29.418412+0000","last_peered":"2024-03-16T20:07:29.418412+0000","last_clean":"2024-03-16T20:07:29.418412+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:07:29.418412+0000","last_undegraded":"2024-03-16T20:07:29.418412+0000","last_fullsized":"2024-03-16T20:07:29.418412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467410+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T20:07:53.467410+0000","last_peered":"2024-03-16T20:07:53.467410+0000","last_clean":"2024-03-16T20:07:53.467410+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:07:53.467410+0000","last_undegraded":"2024-03-16T20:07:53.467410+0000","last_fullsized":"2024-03-16T20:07:53.467410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421807+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T20:07:54.421807+0000","last_peered":"2024-03-16T20:07:54.421807+0000","last_clean":"2024-03-16T20:07:54.421807+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:07:54.421807+0000","last_undegraded":"2024-03-16T20:07:54.421807+0000","last_fullsized":"2024-03-16T20:07:54.421807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.399380+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T20:07:53.399380+0000","last_peered":"2024-03-16T20:07:53.399380+0000","last_clean":"2024-03-16T20:07:53.399380+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:07:53.399380+0000","last_undegraded":"2024-03-16T20:07:53.399380+0000","last_fullsized":"2024-03-16T20:07:53.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.452068+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T20:07:54.452068+0000","last_peered":"2024-03-16T20:07:54.452068+0000","last_clean":"2024-03-16T20:07:54.452068+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:07:54.452068+0000","last_undegraded":"2024-03-16T20:07:54.452068+0000","last_fullsized":"2024-03-16T20:07:54.452068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396659+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T20:07:54.396659+0000","last_peered":"2024-03-16T20:07:54.396659+0000","last_clean":"2024-03-16T20:07:54.396659+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:07:54.396659+0000","last_undegraded":"2024-03-16T20:07:54.396659+0000","last_fullsized":"2024-03-16T20:07:54.396659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.415607+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T20:08:18.415607+0000","last_peered":"2024-03-16T20:08:18.415607+0000","last_clean":"2024-03-16T20:08:18.415607+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:08:18.415607+0000","last_undegraded":"2024-03-16T20:08:18.415607+0000","last_fullsized":"2024-03-16T20:08:18.415607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425243+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T20:08:19.425243+0000","last_peered":"2024-03-16T20:08:19.425243+0000","last_clean":"2024-03-16T20:08:19.425243+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:08:19.425243+0000","last_undegraded":"2024-03-16T20:08:19.425243+0000","last_fullsized":"2024-03-16T20:08:19.425243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425164+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T20:08:19.425164+0000","last_peered":"2024-03-16T20:08:19.425164+0000","last_clean":"2024-03-16T20:08:19.425164+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:08:19.425164+0000","last_undegraded":"2024-03-16T20:08:19.425164+0000","last_fullsized":"2024-03-16T20:08:19.425164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:15.641445+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T20:08:15.641445+0000","last_peered":"2024-03-16T20:08:15.641445+0000","last_clean":"2024-03-16T20:08:15.641445+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:08:15.641445+0000","last_undegraded":"2024-03-16T20:08:15.641445+0000","last_fullsized":"2024-03-16T20:08:15.641445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.401956+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T20:08:18.401956+0000","last_peered":"2024-03-16T20:08:18.401956+0000","last_clean":"2024-03-16T20:08:18.401956+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:08:18.401956+0000","last_undegraded":"2024-03-16T20:08:18.401956+0000","last_fullsized":"2024-03-16T20:08:18.401956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473580+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T20:08:43.473580+0000","last_peered":"2024-03-16T20:08:43.473580+0000","last_clean":"2024-03-16T20:08:43.473580+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:08:43.473580+0000","last_undegraded":"2024-03-16T20:08:43.473580+0000","last_fullsized":"2024-03-16T20:08:43.473580+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.478645+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T20:08:44.478645+0000","last_peered":"2024-03-16T20:08:44.478645+0000","last_clean":"2024-03-16T20:08:44.478645+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:08:44.478645+0000","last_undegraded":"2024-03-16T20:08:44.478645+0000","last_fullsized":"2024-03-16T20:08:44.478645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457937+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T20:08:44.457937+0000","last_peered":"2024-03-16T20:08:44.457937+0000","last_clean":"2024-03-16T20:08:44.457937+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:08:44.457937+0000","last_undegraded":"2024-03-16T20:08:44.457937+0000","last_fullsized":"2024-03-16T20:08:44.457937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457968+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T20:08:44.457968+0000","last_peered":"2024-03-16T20:08:44.457968+0000","last_clean":"2024-03-16T20:08:44.457968+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:08:44.457968+0000","last_undegraded":"2024-03-16T20:08:44.457968+0000","last_fullsized":"2024-03-16T20:08:44.457968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402472+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T20:08:44.402472+0000","last_peered":"2024-03-16T20:08:44.402472+0000","last_clean":"2024-03-16T20:08:44.402472+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:08:44.402472+0000","last_undegraded":"2024-03-16T20:08:44.402472+0000","last_fullsized":"2024-03-16T20:08:44.402472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:38.457785+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T20:06:38.457785+0000","last_peered":"2024-03-16T20:06:38.457785+0000","last_clean":"2024-03-16T20:06:38.457785+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:06:38.457785+0000","last_undegraded":"2024-03-16T20:06:38.457785+0000","last_fullsized":"2024-03-16T20:06:38.457785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387104+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T20:06:39.387104+0000","last_peered":"2024-03-16T20:06:39.387104+0000","last_clean":"2024-03-16T20:06:39.387104+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:06:39.387104+0000","last_undegraded":"2024-03-16T20:06:39.387104+0000","last_fullsized":"2024-03-16T20:06:39.387104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411230+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T20:06:39.411230+0000","last_peered":"2024-03-16T20:06:39.411230+0000","last_clean":"2024-03-16T20:06:39.411230+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:06:39.411230+0000","last_undegraded":"2024-03-16T20:06:39.411230+0000","last_fullsized":"2024-03-16T20:06:39.411230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387143+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T20:06:39.387143+0000","last_peered":"2024-03-16T20:06:39.387143+0000","last_clean":"2024-03-16T20:06:39.387143+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:06:39.387143+0000","last_undegraded":"2024-03-16T20:06:39.387143+0000","last_fullsized":"2024-03-16T20:06:39.387143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:43.411988+0000","last_change":"2024-03-16T19:57:41.236414+0000","last_active":"2024-03-16T20:07:43.411988+0000","last_peered":"2024-03-16T20:07:43.411988+0000","last_clean":"2024-03-16T20:07:43.411988+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:07:43.411988+0000","last_undegraded":"2024-03-16T20:07:43.411988+0000","last_fullsized":"2024-03-16T20:07:43.411988+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:49.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408280+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T20:09:08.408280+0000","last_peered":"2024-03-16T20:09:08.408280+0000","last_clean":"2024-03-16T20:09:08.408280+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:09:08.408280+0000","last_undegraded":"2024-03-16T20:09:08.408280+0000","last_fullsized":"2024-03-16T20:09:08.408280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647723+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T20:09:05.647723+0000","last_peered":"2024-03-16T20:09:05.647723+0000","last_clean":"2024-03-16T20:09:05.647723+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:09:05.647723+0000","last_undegraded":"2024-03-16T20:09:05.647723+0000","last_fullsized":"2024-03-16T20:09:05.647723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405802+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T20:09:09.405802+0000","last_peered":"2024-03-16T20:09:09.405802+0000","last_clean":"2024-03-16T20:09:09.405802+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:09:09.405802+0000","last_undegraded":"2024-03-16T20:09:09.405802+0000","last_fullsized":"2024-03-16T20:09:09.405802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476929,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476931,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476930,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476928,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509631,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509633,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509633,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542337,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:11:38.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:11:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 Processor -- start 2024-03-16T20:11:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 -- start start 2024-03-16T20:11:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24640b1eb0 0x7f2464072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f24640c7ce0 0x7f2464072f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640c68e0 con 0x7f246406ea80 2024-03-16T20:11:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2464067340 con 0x7f24640b1eb0 2024-03-16T20:11:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.476+0000 7f24694a9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246406c1f0 con 0x7f24640c7ce0 2024-03-16T20:11:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.480+0000 7f2462ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24640b1eb0 0x7f2464072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.480+0000 7f24627fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f24640c7ce0 0x7f2464072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.480+0000 7f2462ffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24640b1eb0 0x7f2464072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49034/0 (socket says 172.21.15.32:49034) 2024-03-16T20:11:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.480+0000 7f2462ffd640 1 -- 172.21.15.32:0/314274991 learned_addr learned my addr 172.21.15.32:0/314274991 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.484+0000 7f2462ffd640 1 -- 172.21.15.32:0/314274991 >> v2:172.21.15.59:3300/0 conn(0x7f24640c7ce0 msgr2=0x7f2464072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.484+0000 7f2462ffd640 1 --2- 172.21.15.32:0/314274991 >> v2:172.21.15.59:3300/0 conn(0x7f24640c7ce0 0x7f2464072f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.484+0000 7f2462ffd640 1 -- 172.21.15.32:0/314274991 >> v1:172.21.15.59:6789/0 conn(0x7f246406ea80 legacy=0x7f2464158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.484+0000 7f2462ffd640 1 -- 172.21.15.32:0/314274991 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246406c4d0 con 0x7f24640b1eb0 2024-03-16T20:11:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.488+0000 7f2462ffd640 1 --2- 172.21.15.32:0/314274991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24640b1eb0 0x7f2464072a10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2464073530 tx=0x7f245401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119ff98474ed4188 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f2461ffb640 1 -- 172.21.15.32:0/314274991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540a89d0 con 0x7f24640b1eb0 2024-03-16T20:11:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f2461ffb640 1 -- 172.21.15.32:0/314274991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540a8b30 con 0x7f24640b1eb0 2024-03-16T20:11:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f2461ffb640 1 -- 172.21.15.32:0/314274991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540b0440 con 0x7f24640b1eb0 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 -- 172.21.15.32:0/314274991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24640b1eb0 msgr2=0x7f2464072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 --2- 172.21.15.32:0/314274991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24640b1eb0 0x7f2464072a10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2464073530 tx=0x7f245401ecd0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 -- 172.21.15.32:0/314274991 shutdown_connections 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 --2- 172.21.15.32:0/314274991 >> v2:172.21.15.59:3300/0 conn(0x7f24640c7ce0 0x7f2464072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 --2- 172.21.15.32:0/314274991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24640b1eb0 0x7f2464072a10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 -- 172.21.15.32:0/314274991 >> 172.21.15.32:0/314274991 conn(0x7f246406afc0 msgr2=0x7f246406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 -- 172.21.15.32:0/314274991 shutdown_connections 2024-03-16T20:11:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.492+0000 7f24694a9640 1 -- 172.21.15.32:0/314274991 wait complete. 2024-03-16T20:11:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 Processor -- start 2024-03-16T20:11:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 -- start start 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24640b1eb0 0x7f24641567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24640c7ce0 0x7f2464156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24641f3da0 con 0x7f246406ea80 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24641efc10 con 0x7f24640c7ce0 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24694a9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246414a370 con 0x7f24640b1eb0 2024-03-16T20:11:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49042/0 (socket says 172.21.15.32:49042) 2024-03-16T20:11:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f2462ffd640 1 -- 172.21.15.32:0/2457800702 learned_addr learned my addr 172.21.15.32:0/2457800702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24627fc640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24640b1eb0 0x7f24641567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.496+0000 7f24637fe640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24640c7ce0 0x7f2464156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 -- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24640b1eb0 msgr2=0x7f24641567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24640b1eb0 0x7f24641567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 -- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24640c7ce0 msgr2=0x7f2464156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24640c7ce0 0x7f2464156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 -- 172.21.15.32:0/2457800702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24540094d0 con 0x7f246406ea80 2024-03-16T20:11:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.504+0000 7f2462ffd640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f245401d380 tx=0x7f245401ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540b0dd0 con 0x7f246406ea80 2024-03-16T20:11:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540bb460 con 0x7f246406ea80 2024-03-16T20:11:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640c0f90 con 0x7f246406ea80 2024-03-16T20:11:38.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540bb810 con 0x7f246406ea80 2024-03-16T20:11:38.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24641ef8c0 con 0x7f246406ea80 2024-03-16T20:11:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.512+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24641efa50 con 0x7f246406ea80 2024-03-16T20:11:38.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.816+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f24540b83f0 con 0x7f246406ea80 2024-03-16T20:11:38.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.816+0000 7f244b7fe640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 0x7f2450069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:38.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.816+0000 7f24627fc640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 0x7f2450069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:38.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.816+0000 7f24627fc640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 0x7f2450069300 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f24640b1a70 tx=0x7f2458009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:38.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.836+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f24540f3480 con 0x7f246406ea80 2024-03-16T20:11:38.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:38.836+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f24540ef720 con 0x7f246406ea80 2024-03-16T20:11:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.068+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24640c5a30 con 0x7f2450066e40 2024-03-16T20:11:39.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.084+0000 7f244b7fe640 1 -- 172.21.15.32:0/2457800702 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332622 (secure 0 0 0) 0x7f24640c5a30 con 0x7f2450066e40 2024-03-16T20:11:39.080 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:11:39.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 msgr2=0x7f2450069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 0x7f2450069300 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f24640b1a70 tx=0x7f2458009190 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 msgr2=0x7f24641f00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f245401d380 tx=0x7f245401ba70 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 shutdown_connections 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2450066e40 0x7f2450069300 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24640c7ce0 0x7f2464156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f24640b1eb0 0x7f24641567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 --2- 172.21.15.32:0/2457800702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f246406ea80 0x7f24641f00a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 >> 172.21.15.32:0/2457800702 conn(0x7f246406afc0 msgr2=0x7f24640c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 shutdown_connections 2024-03-16T20:11:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.088+0000 7f24694a9640 1 -- 172.21.15.32:0/2457800702 wait complete. 2024-03-16T20:11:39.098 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-03-16T20:11:38.846251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236864,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737856,"statfs":{"total":773094113280,"available":772851564544,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003550"},"pg_stats":[{"pgid":"2.1f","version":"32'14","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.407709+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T20:09:03.407709+0000","last_peered":"2024-03-16T20:09:03.407709+0000","last_clean":"2024-03-16T20:09:03.407709+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:09:03.407709+0000","last_undegraded":"2024-03-16T20:09:03.407709+0000","last_fullsized":"2024-03-16T20:09:03.407709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.473025+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T20:08:38.473025+0000","last_peered":"2024-03-16T20:08:38.473025+0000","last_clean":"2024-03-16T20:08:38.473025+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:08:38.473025+0000","last_undegraded":"2024-03-16T20:08:38.473025+0000","last_fullsized":"2024-03-16T20:08:38.473025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474069+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T20:08:48.474069+0000","last_peered":"2024-03-16T20:08:48.474069+0000","last_clean":"2024-03-16T20:08:48.474069+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:08:48.474069+0000","last_undegraded":"2024-03-16T20:08:48.474069+0000","last_fullsized":"2024-03-16T20:08:48.474069+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.472417+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T20:08:33.472417+0000","last_peered":"2024-03-16T20:08:33.472417+0000","last_clean":"2024-03-16T20:08:33.472417+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:08:33.472417+0000","last_undegraded":"2024-03-16T20:08:33.472417+0000","last_fullsized":"2024-03-16T20:08:33.472417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408303+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T20:09:08.408303+0000","last_peered":"2024-03-16T20:09:08.408303+0000","last_clean":"2024-03-16T20:09:08.408303+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:09:08.408303+0000","last_undegraded":"2024-03-16T20:09:08.408303+0000","last_fullsized":"2024-03-16T20:09:08.408303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.479106+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T20:08:49.479106+0000","last_peered":"2024-03-16T20:08:49.479106+0000","last_clean":"2024-03-16T20:08:49.479106+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:08:49.479106+0000","last_undegraded":"2024-03-16T20:08:49.479106+0000","last_fullsized":"2024-03-16T20:08:49.479106+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'13","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405198+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T20:09:04.405198+0000","last_peered":"2024-03-16T20:09:04.405198+0000","last_clean":"2024-03-16T20:09:04.405198+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:09:04.405198+0000","last_undegraded":"2024-03-16T20:09:04.405198+0000","last_fullsized":"2024-03-16T20:09:04.405198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411271+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T20:06:39.411271+0000","last_peered":"2024-03-16T20:06:39.411271+0000","last_clean":"2024-03-16T20:06:39.411271+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:06:39.411271+0000","last_undegraded":"2024-03-16T20:06:39.411271+0000","last_fullsized":"2024-03-16T20:06:39.411271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464297+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T20:06:44.464297+0000","last_peered":"2024-03-16T20:06:44.464297+0000","last_clean":"2024-03-16T20:06:44.464297+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:06:44.464297+0000","last_undegraded":"2024-03-16T20:06:44.464297+0000","last_fullsized":"2024-03-16T20:06:44.464297+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405162+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T20:09:04.405162+0000","last_peered":"2024-03-16T20:09:04.405162+0000","last_clean":"2024-03-16T20:09:04.405162+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:09:04.405162+0000","last_undegraded":"2024-03-16T20:09:04.405162+0000","last_fullsized":"2024-03-16T20:09:04.405162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411290+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T20:06:39.411290+0000","last_peered":"2024-03-16T20:06:39.411290+0000","last_clean":"2024-03-16T20:06:39.411290+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:06:39.411290+0000","last_undegraded":"2024-03-16T20:06:39.411290+0000","last_fullsized":"2024-03-16T20:06:39.411290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464225+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T20:06:44.464225+0000","last_peered":"2024-03-16T20:06:44.464225+0000","last_clean":"2024-03-16T20:06:44.464225+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:06:44.464225+0000","last_undegraded":"2024-03-16T20:06:44.464225+0000","last_fullsized":"2024-03-16T20:06:44.464225+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448622+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T20:07:29.448622+0000","last_peered":"2024-03-16T20:07:29.448622+0000","last_clean":"2024-03-16T20:07:29.448622+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:07:29.448622+0000","last_undegraded":"2024-03-16T20:07:29.448622+0000","last_fullsized":"2024-03-16T20:07:29.448622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.395875+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T20:07:49.395875+0000","last_peered":"2024-03-16T20:07:49.395875+0000","last_clean":"2024-03-16T20:07:49.395875+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:07:49.395875+0000","last_undegraded":"2024-03-16T20:07:49.395875+0000","last_fullsized":"2024-03-16T20:07:49.395875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473222+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T20:07:59.473222+0000","last_peered":"2024-03-16T20:07:59.473222+0000","last_clean":"2024-03-16T20:07:59.473222+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:07:59.473222+0000","last_undegraded":"2024-03-16T20:07:59.473222+0000","last_fullsized":"2024-03-16T20:07:59.473222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467453+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T20:07:53.467453+0000","last_peered":"2024-03-16T20:07:53.467453+0000","last_clean":"2024-03-16T20:07:53.467453+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:07:53.467453+0000","last_undegraded":"2024-03-16T20:07:53.467453+0000","last_fullsized":"2024-03-16T20:07:53.467453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635909+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T20:07:25.635909+0000","last_peered":"2024-03-16T20:07:25.635909+0000","last_clean":"2024-03-16T20:07:25.635909+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:07:25.635909+0000","last_undegraded":"2024-03-16T20:07:25.635909+0000","last_fullsized":"2024-03-16T20:07:25.635909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473182+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T20:07:59.473182+0000","last_peered":"2024-03-16T20:07:59.473182+0000","last_clean":"2024-03-16T20:07:59.473182+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:07:59.473182+0000","last_undegraded":"2024-03-16T20:07:59.473182+0000","last_fullsized":"2024-03-16T20:07:59.473182+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463912+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T20:07:28.463912+0000","last_peered":"2024-03-16T20:07:28.463912+0000","last_clean":"2024-03-16T20:07:28.463912+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:07:28.463912+0000","last_undegraded":"2024-03-16T20:07:28.463912+0000","last_fullsized":"2024-03-16T20:07:28.463912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635877+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T20:07:25.635877+0000","last_peered":"2024-03-16T20:07:25.635877+0000","last_clean":"2024-03-16T20:07:25.635877+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:07:25.635877+0000","last_undegraded":"2024-03-16T20:07:25.635877+0000","last_fullsized":"2024-03-16T20:07:25.635877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:00.639712+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T20:08:00.639712+0000","last_peered":"2024-03-16T20:08:00.639712+0000","last_clean":"2024-03-16T20:08:00.639712+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:08:00.639712+0000","last_undegraded":"2024-03-16T20:08:00.639712+0000","last_fullsized":"2024-03-16T20:08:00.639712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.469295+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T20:07:29.469295+0000","last_peered":"2024-03-16T20:07:29.469295+0000","last_clean":"2024-03-16T20:07:29.469295+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:07:29.469295+0000","last_undegraded":"2024-03-16T20:07:29.469295+0000","last_fullsized":"2024-03-16T20:07:29.469295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.392918+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T20:07:24.392918+0000","last_peered":"2024-03-16T20:07:24.392918+0000","last_clean":"2024-03-16T20:07:24.392918+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:07:24.392918+0000","last_undegraded":"2024-03-16T20:07:24.392918+0000","last_fullsized":"2024-03-16T20:07:24.392918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473238+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T20:07:59.473238+0000","last_peered":"2024-03-16T20:07:59.473238+0000","last_clean":"2024-03-16T20:07:59.473238+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:07:59.473238+0000","last_undegraded":"2024-03-16T20:07:59.473238+0000","last_fullsized":"2024-03-16T20:07:59.473238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.395991+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T20:07:28.395991+0000","last_peered":"2024-03-16T20:07:28.395991+0000","last_clean":"2024-03-16T20:07:28.395991+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:07:28.395991+0000","last_undegraded":"2024-03-16T20:07:28.395991+0000","last_fullsized":"2024-03-16T20:07:28.395991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.448140+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T20:07:24.448140+0000","last_peered":"2024-03-16T20:07:24.448140+0000","last_clean":"2024-03-16T20:07:24.448140+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:07:24.448140+0000","last_undegraded":"2024-03-16T20:07:24.448140+0000","last_fullsized":"2024-03-16T20:07:24.448140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.481902+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T20:09:14.481902+0000","last_peered":"2024-03-16T20:09:14.481902+0000","last_clean":"2024-03-16T20:09:14.481902+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:09:14.481902+0000","last_undegraded":"2024-03-16T20:09:14.481902+0000","last_fullsized":"2024-03-16T20:09:14.481902+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.417264+0000","last_change":"2024-03-16T19:57:41.287408+0000","last_active":"2024-03-16T20:08:33.417264+0000","last_peered":"2024-03-16T20:08:33.417264+0000","last_clean":"2024-03-16T20:08:33.417264+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:08:33.417264+0000","last_undegraded":"2024-03-16T20:08:33.417264+0000","last_fullsized":"2024-03-16T20:08:33.417264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:54.981214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635961+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T20:07:25.635961+0000","last_peered":"2024-03-16T20:07:25.635961+0000","last_clean":"2024-03-16T20:07:25.635961+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:07:25.635961+0000","last_undegraded":"2024-03-16T20:07:25.635961+0000","last_fullsized":"2024-03-16T20:07:25.635961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473514+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T20:08:43.473514+0000","last_peered":"2024-03-16T20:08:43.473514+0000","last_clean":"2024-03-16T20:08:43.473514+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:08:43.473514+0000","last_undegraded":"2024-03-16T20:08:43.473514+0000","last_fullsized":"2024-03-16T20:08:43.473514+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647671+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T20:09:05.647671+0000","last_peered":"2024-03-16T20:09:05.647671+0000","last_clean":"2024-03-16T20:09:05.647671+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:09:05.647671+0000","last_undegraded":"2024-03-16T20:09:05.647671+0000","last_fullsized":"2024-03-16T20:09:05.647671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405217+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T20:09:04.405217+0000","last_peered":"2024-03-16T20:09:04.405217+0000","last_clean":"2024-03-16T20:09:04.405217+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:09:04.405217+0000","last_undegraded":"2024-03-16T20:09:04.405217+0000","last_fullsized":"2024-03-16T20:09:04.405217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:13.408811+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T20:09:13.408811+0000","last_peered":"2024-03-16T20:09:13.408811+0000","last_clean":"2024-03-16T20:09:13.408811+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:09:13.408811+0000","last_undegraded":"2024-03-16T20:09:13.408811+0000","last_fullsized":"2024-03-16T20:09:13.408811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432177+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T20:09:09.432177+0000","last_peered":"2024-03-16T20:09:09.432177+0000","last_clean":"2024-03-16T20:09:09.432177+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:09:09.432177+0000","last_undegraded":"2024-03-16T20:09:09.432177+0000","last_fullsized":"2024-03-16T20:09:09.432177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647700+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T20:09:05.647700+0000","last_peered":"2024-03-16T20:09:05.647700+0000","last_clean":"2024-03-16T20:09:05.647700+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:09:05.647700+0000","last_undegraded":"2024-03-16T20:09:05.647700+0000","last_fullsized":"2024-03-16T20:09:05.647700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648736+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T20:09:15.648736+0000","last_peered":"2024-03-16T20:09:15.648736+0000","last_clean":"2024-03-16T20:09:15.648736+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:09:15.648736+0000","last_undegraded":"2024-03-16T20:09:15.648736+0000","last_fullsized":"2024-03-16T20:09:15.648736+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.481151+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T20:09:09.481151+0000","last_peered":"2024-03-16T20:09:09.481151+0000","last_clean":"2024-03-16T20:09:09.481151+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:09:09.481151+0000","last_undegraded":"2024-03-16T20:09:09.481151+0000","last_fullsized":"2024-03-16T20:09:09.481151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'6","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431117+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T20:09:04.431117+0000","last_peered":"2024-03-16T20:09:04.431117+0000","last_clean":"2024-03-16T20:09:04.431117+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:09:04.431117+0000","last_undegraded":"2024-03-16T20:09:04.431117+0000","last_fullsized":"2024-03-16T20:09:04.431117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648771+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T20:09:15.648771+0000","last_peered":"2024-03-16T20:09:15.648771+0000","last_clean":"2024-03-16T20:09:15.648771+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:09:15.648771+0000","last_undegraded":"2024-03-16T20:09:15.648771+0000","last_fullsized":"2024-03-16T20:09:15.648771+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:35.643746+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T20:08:35.643746+0000","last_peered":"2024-03-16T20:08:35.643746+0000","last_clean":"2024-03-16T20:08:35.643746+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:08:35.643746+0000","last_undegraded":"2024-03-16T20:08:35.643746+0000","last_fullsized":"2024-03-16T20:08:35.643746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.420300+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T20:09:03.420300+0000","last_peered":"2024-03-16T20:09:03.420300+0000","last_clean":"2024-03-16T20:09:03.420300+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:09:03.420300+0000","last_undegraded":"2024-03-16T20:09:03.420300+0000","last_fullsized":"2024-03-16T20:09:03.420300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:35.636979+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T20:07:35.636979+0000","last_peered":"2024-03-16T20:07:35.636979+0000","last_clean":"2024-03-16T20:07:35.636979+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:07:35.636979+0000","last_undegraded":"2024-03-16T20:07:35.636979+0000","last_fullsized":"2024-03-16T20:07:35.636979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408207+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T20:09:08.408207+0000","last_peered":"2024-03-16T20:09:08.408207+0000","last_clean":"2024-03-16T20:09:08.408207+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:09:08.408207+0000","last_undegraded":"2024-03-16T20:09:08.408207+0000","last_fullsized":"2024-03-16T20:09:08.408207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:34.401181+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T20:08:34.401181+0000","last_peered":"2024-03-16T20:08:34.401181+0000","last_clean":"2024-03-16T20:08:34.401181+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:08:34.401181+0000","last_undegraded":"2024-03-16T20:08:34.401181+0000","last_fullsized":"2024-03-16T20:08:34.401181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:33.396597+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T20:07:33.396597+0000","last_peered":"2024-03-16T20:07:33.396597+0000","last_clean":"2024-03-16T20:07:33.396597+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:07:33.396597+0000","last_undegraded":"2024-03-16T20:07:33.396597+0000","last_fullsized":"2024-03-16T20:07:33.396597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.476309+0000","last_change":"2024-03-16T19:57:41.238831+0000","last_active":"2024-03-16T20:09:08.476309+0000","last_peered":"2024-03-16T20:09:08.476309+0000","last_clean":"2024-03-16T20:09:08.476309+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:09:08.476309+0000","last_undegraded":"2024-03-16T20:09:08.476309+0000","last_fullsized":"2024-03-16T20:09:08.476309+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:56:28.075437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431163+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T20:09:04.431163+0000","last_peered":"2024-03-16T20:09:04.431163+0000","last_clean":"2024-03-16T20:09:04.431163+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:09:04.431163+0000","last_undegraded":"2024-03-16T20:09:04.431163+0000","last_fullsized":"2024-03-16T20:09:04.431163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":197,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396571+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T20:07:54.396571+0000","last_peered":"2024-03-16T20:07:54.396571+0000","last_clean":"2024-03-16T20:07:54.396571+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:07:54.396571+0000","last_undegraded":"2024-03-16T20:07:54.396571+0000","last_fullsized":"2024-03-16T20:07:54.396571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461272+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T20:09:14.461272+0000","last_peered":"2024-03-16T20:09:14.461272+0000","last_clean":"2024-03-16T20:09:14.461272+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:09:14.461272+0000","last_undegraded":"2024-03-16T20:09:14.461272+0000","last_fullsized":"2024-03-16T20:09:14.461272+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396635+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T20:07:54.396635+0000","last_peered":"2024-03-16T20:07:54.396635+0000","last_clean":"2024-03-16T20:07:54.396635+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:07:54.396635+0000","last_undegraded":"2024-03-16T20:07:54.396635+0000","last_fullsized":"2024-03-16T20:07:54.396635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412513+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T20:07:48.412513+0000","last_peered":"2024-03-16T20:07:48.412513+0000","last_clean":"2024-03-16T20:07:48.412513+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:07:48.412513+0000","last_undegraded":"2024-03-16T20:07:48.412513+0000","last_fullsized":"2024-03-16T20:07:48.412513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.449341+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T20:07:34.449341+0000","last_peered":"2024-03-16T20:07:34.449341+0000","last_clean":"2024-03-16T20:07:34.449341+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:07:34.449341+0000","last_undegraded":"2024-03-16T20:07:34.449341+0000","last_fullsized":"2024-03-16T20:07:34.449341+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421860+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T20:07:54.421860+0000","last_peered":"2024-03-16T20:07:54.421860+0000","last_clean":"2024-03-16T20:07:54.421860+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:07:54.421860+0000","last_undegraded":"2024-03-16T20:07:54.421860+0000","last_fullsized":"2024-03-16T20:07:54.421860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:50.638775+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T20:07:50.638775+0000","last_peered":"2024-03-16T20:07:50.638775+0000","last_clean":"2024-03-16T20:07:50.638775+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:07:50.638775+0000","last_undegraded":"2024-03-16T20:07:50.638775+0000","last_fullsized":"2024-03-16T20:07:50.638775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:58.399907+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T20:07:58.399907+0000","last_peered":"2024-03-16T20:07:58.399907+0000","last_clean":"2024-03-16T20:07:58.399907+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:07:58.399907+0000","last_undegraded":"2024-03-16T20:07:58.399907+0000","last_fullsized":"2024-03-16T20:07:58.399907+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396733+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T20:07:54.396733+0000","last_peered":"2024-03-16T20:07:54.396733+0000","last_clean":"2024-03-16T20:07:54.396733+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:07:54.396733+0000","last_undegraded":"2024-03-16T20:07:54.396733+0000","last_fullsized":"2024-03-16T20:07:54.396733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412548+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T20:07:48.412548+0000","last_peered":"2024-03-16T20:07:48.412548+0000","last_clean":"2024-03-16T20:07:48.412548+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:07:48.412548+0000","last_undegraded":"2024-03-16T20:07:48.412548+0000","last_fullsized":"2024-03-16T20:07:48.412548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.419200+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T20:07:34.419200+0000","last_peered":"2024-03-16T20:07:34.419200+0000","last_clean":"2024-03-16T20:07:34.419200+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:07:34.419200+0000","last_undegraded":"2024-03-16T20:07:34.419200+0000","last_fullsized":"2024-03-16T20:07:34.419200+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396681+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T20:07:54.396681+0000","last_peered":"2024-03-16T20:07:54.396681+0000","last_clean":"2024-03-16T20:07:54.396681+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:07:54.396681+0000","last_undegraded":"2024-03-16T20:07:54.396681+0000","last_fullsized":"2024-03-16T20:07:54.396681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.471915+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T20:07:49.471915+0000","last_peered":"2024-03-16T20:07:49.471915+0000","last_clean":"2024-03-16T20:07:49.471915+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:07:49.471915+0000","last_undegraded":"2024-03-16T20:07:49.471915+0000","last_fullsized":"2024-03-16T20:07:49.471915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.469980+0000","last_change":"2024-03-16T19:57:41.242958+0000","last_active":"2024-03-16T20:07:34.469980+0000","last_peered":"2024-03-16T20:07:34.469980+0000","last_clean":"2024-03-16T20:07:34.469980+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:07:34.469980+0000","last_undegraded":"2024-03-16T20:07:34.469980+0000","last_fullsized":"2024-03-16T20:07:34.469980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:17:20.536712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399350+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T20:08:19.399350+0000","last_peered":"2024-03-16T20:08:19.399350+0000","last_clean":"2024-03-16T20:08:19.399350+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:08:19.399350+0000","last_undegraded":"2024-03-16T20:08:19.399350+0000","last_fullsized":"2024-03-16T20:08:19.399350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'18","reported_seq":60,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.415178+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T20:08:13.415178+0000","last_peered":"2024-03-16T20:08:13.415178+0000","last_clean":"2024-03-16T20:08:13.415178+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:08:13.415178+0000","last_undegraded":"2024-03-16T20:08:13.415178+0000","last_fullsized":"2024-03-16T20:08:13.415178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.406062+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T20:08:48.406062+0000","last_peered":"2024-03-16T20:08:48.406062+0000","last_clean":"2024-03-16T20:08:48.406062+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:08:48.406062+0000","last_undegraded":"2024-03-16T20:08:48.406062+0000","last_fullsized":"2024-03-16T20:08:48.406062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.455260+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T20:08:19.455260+0000","last_peered":"2024-03-16T20:08:19.455260+0000","last_clean":"2024-03-16T20:08:19.455260+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:08:19.455260+0000","last_undegraded":"2024-03-16T20:08:19.455260+0000","last_fullsized":"2024-03-16T20:08:19.455260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'6","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454559+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T20:08:14.454559+0000","last_peered":"2024-03-16T20:08:14.454559+0000","last_clean":"2024-03-16T20:08:14.454559+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:08:14.454559+0000","last_undegraded":"2024-03-16T20:08:14.454559+0000","last_fullsized":"2024-03-16T20:08:14.454559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474130+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T20:08:48.474130+0000","last_peered":"2024-03-16T20:08:48.474130+0000","last_clean":"2024-03-16T20:08:48.474130+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:08:48.474130+0000","last_undegraded":"2024-03-16T20:08:48.474130+0000","last_fullsized":"2024-03-16T20:08:48.474130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399380+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T20:08:19.399380+0000","last_peered":"2024-03-16T20:08:19.399380+0000","last_clean":"2024-03-16T20:08:19.399380+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:08:19.399380+0000","last_undegraded":"2024-03-16T20:08:19.399380+0000","last_fullsized":"2024-03-16T20:08:19.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.424455+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T20:08:14.424455+0000","last_peered":"2024-03-16T20:08:14.424455+0000","last_clean":"2024-03-16T20:08:14.424455+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:08:14.424455+0000","last_undegraded":"2024-03-16T20:08:14.424455+0000","last_fullsized":"2024-03-16T20:08:14.424455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470857+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T20:08:23.470857+0000","last_peered":"2024-03-16T20:08:23.470857+0000","last_clean":"2024-03-16T20:08:23.470857+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:08:23.470857+0000","last_undegraded":"2024-03-16T20:08:23.470857+0000","last_fullsized":"2024-03-16T20:08:23.470857+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:03.407915+0000","last_change":"2024-03-16T19:57:41.287279+0000","last_active":"2024-03-16T20:07:03.407915+0000","last_peered":"2024-03-16T20:07:03.407915+0000","last_clean":"2024-03-16T20:07:03.407915+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:07:03.407915+0000","last_undegraded":"2024-03-16T20:07:03.407915+0000","last_fullsized":"2024-03-16T20:07:03.407915+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:46.850016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.401433+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T20:08:13.401433+0000","last_peered":"2024-03-16T20:08:13.401433+0000","last_clean":"2024-03-16T20:08:13.401433+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:08:13.401433+0000","last_undegraded":"2024-03-16T20:08:13.401433+0000","last_fullsized":"2024-03-16T20:08:13.401433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.429172+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T20:08:49.429172+0000","last_peered":"2024-03-16T20:08:49.429172+0000","last_clean":"2024-03-16T20:08:49.429172+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:08:49.429172+0000","last_undegraded":"2024-03-16T20:08:49.429172+0000","last_fullsized":"2024-03-16T20:08:49.429172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402419+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T20:08:44.402419+0000","last_peered":"2024-03-16T20:08:44.402419+0000","last_clean":"2024-03-16T20:08:44.402419+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:08:44.402419+0000","last_undegraded":"2024-03-16T20:08:44.402419+0000","last_fullsized":"2024-03-16T20:08:44.402419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454598+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T20:08:14.454598+0000","last_peered":"2024-03-16T20:08:14.454598+0000","last_clean":"2024-03-16T20:08:14.454598+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:08:14.454598+0000","last_undegraded":"2024-03-16T20:08:14.454598+0000","last_fullsized":"2024-03-16T20:08:14.454598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:25.642519+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T20:08:25.642519+0000","last_peered":"2024-03-16T20:08:25.642519+0000","last_clean":"2024-03-16T20:08:25.642519+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:08:25.642519+0000","last_undegraded":"2024-03-16T20:08:25.642519+0000","last_fullsized":"2024-03-16T20:08:25.642519+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.475987+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T20:08:19.475987+0000","last_peered":"2024-03-16T20:08:19.475987+0000","last_clean":"2024-03-16T20:08:19.475987+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:08:19.475987+0000","last_undegraded":"2024-03-16T20:08:19.475987+0000","last_fullsized":"2024-03-16T20:08:19.475987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644613+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T20:08:40.644613+0000","last_peered":"2024-03-16T20:08:40.644613+0000","last_clean":"2024-03-16T20:08:40.644613+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:08:40.644613+0000","last_undegraded":"2024-03-16T20:08:40.644613+0000","last_fullsized":"2024-03-16T20:08:40.644613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:24.476654+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T20:08:24.476654+0000","last_peered":"2024-03-16T20:08:24.476654+0000","last_clean":"2024-03-16T20:08:24.476654+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:08:24.476654+0000","last_undegraded":"2024-03-16T20:08:24.476654+0000","last_fullsized":"2024-03-16T20:08:24.476654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405275+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T20:08:43.405275+0000","last_peered":"2024-03-16T20:08:43.405275+0000","last_clean":"2024-03-16T20:08:43.405275+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:08:43.405275+0000","last_undegraded":"2024-03-16T20:08:43.405275+0000","last_fullsized":"2024-03-16T20:08:43.405275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427645+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T20:08:39.427645+0000","last_peered":"2024-03-16T20:08:39.427645+0000","last_clean":"2024-03-16T20:08:39.427645+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:08:39.427645+0000","last_undegraded":"2024-03-16T20:08:39.427645+0000","last_fullsized":"2024-03-16T20:08:39.427645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470727+0000","last_change":"2024-03-16T19:57:41.246101+0000","last_active":"2024-03-16T20:08:23.470727+0000","last_peered":"2024-03-16T20:08:23.470727+0000","last_clean":"2024-03-16T20:08:23.470727+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:08:23.470727+0000","last_undegraded":"2024-03-16T20:08:23.470727+0000","last_fullsized":"2024-03-16T20:08:23.470727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:04.608965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.428389+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T20:08:44.428389+0000","last_peered":"2024-03-16T20:08:44.428389+0000","last_clean":"2024-03-16T20:08:44.428389+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:08:44.428389+0000","last_undegraded":"2024-03-16T20:08:44.428389+0000","last_fullsized":"2024-03-16T20:08:44.428389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.457446+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T20:08:39.457446+0000","last_peered":"2024-03-16T20:08:39.457446+0000","last_clean":"2024-03-16T20:08:39.457446+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:08:39.457446+0000","last_undegraded":"2024-03-16T20:08:39.457446+0000","last_fullsized":"2024-03-16T20:08:39.457446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470791+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T20:08:23.470791+0000","last_peered":"2024-03-16T20:08:23.470791+0000","last_clean":"2024-03-16T20:08:23.470791+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:08:23.470791+0000","last_undegraded":"2024-03-16T20:08:23.470791+0000","last_fullsized":"2024-03-16T20:08:23.470791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405199+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T20:08:43.405199+0000","last_peered":"2024-03-16T20:08:43.405199+0000","last_clean":"2024-03-16T20:08:43.405199+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:08:43.405199+0000","last_undegraded":"2024-03-16T20:08:43.405199+0000","last_fullsized":"2024-03-16T20:08:43.405199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644545+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T20:08:40.644545+0000","last_peered":"2024-03-16T20:08:40.644545+0000","last_clean":"2024-03-16T20:08:40.644545+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:08:40.644545+0000","last_undegraded":"2024-03-16T20:08:40.644545+0000","last_fullsized":"2024-03-16T20:08:40.644545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427570+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T20:08:39.427570+0000","last_peered":"2024-03-16T20:08:39.427570+0000","last_clean":"2024-03-16T20:08:39.427570+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:08:39.427570+0000","last_undegraded":"2024-03-16T20:08:39.427570+0000","last_fullsized":"2024-03-16T20:08:39.427570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461319+0000","last_change":"2024-03-16T19:57:41.279710+0000","last_active":"2024-03-16T20:09:14.461319+0000","last_peered":"2024-03-16T20:09:14.461319+0000","last_clean":"2024-03-16T20:09:14.461319+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:09:14.461319+0000","last_undegraded":"2024-03-16T20:09:14.461319+0000","last_fullsized":"2024-03-16T20:09:14.461319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:02:54.725387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473548+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T20:08:43.473548+0000","last_peered":"2024-03-16T20:08:43.473548+0000","last_clean":"2024-03-16T20:08:43.473548+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:08:43.473548+0000","last_undegraded":"2024-03-16T20:08:43.473548+0000","last_fullsized":"2024-03-16T20:08:43.473548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.472974+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T20:08:38.472974+0000","last_peered":"2024-03-16T20:08:38.472974+0000","last_clean":"2024-03-16T20:08:38.472974+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:08:38.472974+0000","last_undegraded":"2024-03-16T20:08:38.472974+0000","last_fullsized":"2024-03-16T20:08:38.472974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.459824+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T20:09:04.459824+0000","last_peered":"2024-03-16T20:09:04.459824+0000","last_clean":"2024-03-16T20:09:04.459824+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:09:04.459824+0000","last_undegraded":"2024-03-16T20:09:04.459824+0000","last_fullsized":"2024-03-16T20:09:04.459824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408266+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T20:09:08.408266+0000","last_peered":"2024-03-16T20:09:08.408266+0000","last_clean":"2024-03-16T20:09:08.408266+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:09:08.408266+0000","last_undegraded":"2024-03-16T20:09:08.408266+0000","last_fullsized":"2024-03-16T20:09:08.408266+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.405682+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T20:06:43.405682+0000","last_peered":"2024-03-16T20:06:43.405682+0000","last_clean":"2024-03-16T20:06:43.405682+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:06:43.405682+0000","last_undegraded":"2024-03-16T20:06:43.405682+0000","last_fullsized":"2024-03-16T20:06:43.405682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.475556+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T20:09:03.475556+0000","last_peered":"2024-03-16T20:09:03.475556+0000","last_clean":"2024-03-16T20:09:03.475556+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:09:03.475556+0000","last_undegraded":"2024-03-16T20:09:03.475556+0000","last_fullsized":"2024-03-16T20:09:03.475556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.463549+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T20:06:39.463549+0000","last_peered":"2024-03-16T20:06:39.463549+0000","last_clean":"2024-03-16T20:06:39.463549+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:06:39.463549+0000","last_undegraded":"2024-03-16T20:06:39.463549+0000","last_fullsized":"2024-03-16T20:06:39.463549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.458436+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T20:06:43.458436+0000","last_peered":"2024-03-16T20:06:43.458436+0000","last_clean":"2024-03-16T20:06:43.458436+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:06:43.458436+0000","last_undegraded":"2024-03-16T20:06:43.458436+0000","last_fullsized":"2024-03-16T20:06:43.458436+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.460475+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T20:09:09.460475+0000","last_peered":"2024-03-16T20:09:09.460475+0000","last_clean":"2024-03-16T20:09:09.460475+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:09:09.460475+0000","last_undegraded":"2024-03-16T20:09:09.460475+0000","last_fullsized":"2024-03-16T20:09:09.460475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.420799+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T20:09:08.420799+0000","last_peered":"2024-03-16T20:09:08.420799+0000","last_clean":"2024-03-16T20:09:08.420799+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:09:08.420799+0000","last_undegraded":"2024-03-16T20:09:08.420799+0000","last_fullsized":"2024-03-16T20:09:08.420799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.431997+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T20:09:09.431997+0000","last_peered":"2024-03-16T20:09:09.431997+0000","last_clean":"2024-03-16T20:09:09.431997+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:09:09.431997+0000","last_undegraded":"2024-03-16T20:09:09.431997+0000","last_fullsized":"2024-03-16T20:09:09.431997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408250+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T20:09:08.408250+0000","last_peered":"2024-03-16T20:09:08.408250+0000","last_clean":"2024-03-16T20:09:08.408250+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:09:08.408250+0000","last_undegraded":"2024-03-16T20:09:08.408250+0000","last_fullsized":"2024-03-16T20:09:08.408250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463977+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T20:07:28.463977+0000","last_peered":"2024-03-16T20:07:28.463977+0000","last_clean":"2024-03-16T20:07:28.463977+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:07:28.463977+0000","last_undegraded":"2024-03-16T20:07:28.463977+0000","last_fullsized":"2024-03-16T20:07:28.463977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635930+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T20:07:25.635930+0000","last_peered":"2024-03-16T20:07:25.635930+0000","last_clean":"2024-03-16T20:07:25.635930+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:07:25.635930+0000","last_undegraded":"2024-03-16T20:07:25.635930+0000","last_fullsized":"2024-03-16T20:07:25.635930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448666+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T20:07:29.448666+0000","last_peered":"2024-03-16T20:07:29.448666+0000","last_clean":"2024-03-16T20:07:29.448666+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:07:29.448666+0000","last_undegraded":"2024-03-16T20:07:29.448666+0000","last_fullsized":"2024-03-16T20:07:29.448666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.464010+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T20:07:28.464010+0000","last_peered":"2024-03-16T20:07:28.464010+0000","last_clean":"2024-03-16T20:07:28.464010+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:07:28.464010+0000","last_undegraded":"2024-03-16T20:07:28.464010+0000","last_fullsized":"2024-03-16T20:07:28.464010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.418412+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T20:07:29.418412+0000","last_peered":"2024-03-16T20:07:29.418412+0000","last_clean":"2024-03-16T20:07:29.418412+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:07:29.418412+0000","last_undegraded":"2024-03-16T20:07:29.418412+0000","last_fullsized":"2024-03-16T20:07:29.418412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467410+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T20:07:53.467410+0000","last_peered":"2024-03-16T20:07:53.467410+0000","last_clean":"2024-03-16T20:07:53.467410+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:07:53.467410+0000","last_undegraded":"2024-03-16T20:07:53.467410+0000","last_fullsized":"2024-03-16T20:07:53.467410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421807+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T20:07:54.421807+0000","last_peered":"2024-03-16T20:07:54.421807+0000","last_clean":"2024-03-16T20:07:54.421807+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:07:54.421807+0000","last_undegraded":"2024-03-16T20:07:54.421807+0000","last_fullsized":"2024-03-16T20:07:54.421807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.399380+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T20:07:53.399380+0000","last_peered":"2024-03-16T20:07:53.399380+0000","last_clean":"2024-03-16T20:07:53.399380+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:07:53.399380+0000","last_undegraded":"2024-03-16T20:07:53.399380+0000","last_fullsized":"2024-03-16T20:07:53.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.452068+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T20:07:54.452068+0000","last_peered":"2024-03-16T20:07:54.452068+0000","last_clean":"2024-03-16T20:07:54.452068+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:07:54.452068+0000","last_undegraded":"2024-03-16T20:07:54.452068+0000","last_fullsized":"2024-03-16T20:07:54.452068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396659+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T20:07:54.396659+0000","last_peered":"2024-03-16T20:07:54.396659+0000","last_clean":"2024-03-16T20:07:54.396659+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:07:54.396659+0000","last_undegraded":"2024-03-16T20:07:54.396659+0000","last_fullsized":"2024-03-16T20:07:54.396659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.415607+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T20:08:18.415607+0000","last_peered":"2024-03-16T20:08:18.415607+0000","last_clean":"2024-03-16T20:08:18.415607+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:08:18.415607+0000","last_undegraded":"2024-03-16T20:08:18.415607+0000","last_fullsized":"2024-03-16T20:08:18.415607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425243+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T20:08:19.425243+0000","last_peered":"2024-03-16T20:08:19.425243+0000","last_clean":"2024-03-16T20:08:19.425243+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:08:19.425243+0000","last_undegraded":"2024-03-16T20:08:19.425243+0000","last_fullsized":"2024-03-16T20:08:19.425243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425164+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T20:08:19.425164+0000","last_peered":"2024-03-16T20:08:19.425164+0000","last_clean":"2024-03-16T20:08:19.425164+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:08:19.425164+0000","last_undegraded":"2024-03-16T20:08:19.425164+0000","last_fullsized":"2024-03-16T20:08:19.425164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:15.641445+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T20:08:15.641445+0000","last_peered":"2024-03-16T20:08:15.641445+0000","last_clean":"2024-03-16T20:08:15.641445+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:08:15.641445+0000","last_undegraded":"2024-03-16T20:08:15.641445+0000","last_fullsized":"2024-03-16T20:08:15.641445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.401956+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T20:08:18.401956+0000","last_peered":"2024-03-16T20:08:18.401956+0000","last_clean":"2024-03-16T20:08:18.401956+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:08:18.401956+0000","last_undegraded":"2024-03-16T20:08:18.401956+0000","last_fullsized":"2024-03-16T20:08:18.401956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473580+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T20:08:43.473580+0000","last_peered":"2024-03-16T20:08:43.473580+0000","last_clean":"2024-03-16T20:08:43.473580+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:08:43.473580+0000","last_undegraded":"2024-03-16T20:08:43.473580+0000","last_fullsized":"2024-03-16T20:08:43.473580+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.478645+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T20:08:44.478645+0000","last_peered":"2024-03-16T20:08:44.478645+0000","last_clean":"2024-03-16T20:08:44.478645+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:08:44.478645+0000","last_undegraded":"2024-03-16T20:08:44.478645+0000","last_fullsized":"2024-03-16T20:08:44.478645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457937+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T20:08:44.457937+0000","last_peered":"2024-03-16T20:08:44.457937+0000","last_clean":"2024-03-16T20:08:44.457937+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:08:44.457937+0000","last_undegraded":"2024-03-16T20:08:44.457937+0000","last_fullsized":"2024-03-16T20:08:44.457937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457968+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T20:08:44.457968+0000","last_peered":"2024-03-16T20:08:44.457968+0000","last_clean":"2024-03-16T20:08:44.457968+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:08:44.457968+0000","last_undegraded":"2024-03-16T20:08:44.457968+0000","last_fullsized":"2024-03-16T20:08:44.457968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402472+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T20:08:44.402472+0000","last_peered":"2024-03-16T20:08:44.402472+0000","last_clean":"2024-03-16T20:08:44.402472+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:08:44.402472+0000","last_undegraded":"2024-03-16T20:08:44.402472+0000","last_fullsized":"2024-03-16T20:08:44.402472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.476255+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T20:09:08.476255+0000","last_peered":"2024-03-16T20:09:08.476255+0000","last_clean":"2024-03-16T20:09:08.476255+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:09:08.476255+0000","last_undegraded":"2024-03-16T20:09:08.476255+0000","last_fullsized":"2024-03-16T20:09:08.476255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387104+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T20:06:39.387104+0000","last_peered":"2024-03-16T20:06:39.387104+0000","last_clean":"2024-03-16T20:06:39.387104+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:06:39.387104+0000","last_undegraded":"2024-03-16T20:06:39.387104+0000","last_fullsized":"2024-03-16T20:06:39.387104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411230+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T20:06:39.411230+0000","last_peered":"2024-03-16T20:06:39.411230+0000","last_clean":"2024-03-16T20:06:39.411230+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:06:39.411230+0000","last_undegraded":"2024-03-16T20:06:39.411230+0000","last_fullsized":"2024-03-16T20:06:39.411230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387143+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T20:06:39.387143+0000","last_peered":"2024-03-16T20:06:39.387143+0000","last_clean":"2024-03-16T20:06:39.387143+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:06:39.387143+0000","last_undegraded":"2024-03-16T20:06:39.387143+0000","last_fullsized":"2024-03-16T20:06:39.387143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:43.411988+0000","last_change":"2024-03-16T19:57:41.236414+0000","last_active":"2024-03-16T20:07:43.411988+0000","last_peered":"2024-03-16T20:07:43.411988+0000","last_clean":"2024-03-16T20:07:43.411988+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:07:43.411988+0000","last_undegraded":"2024-03-16T20:07:43.411988+0000","last_fullsized":"2024-03-16T20:07:43.411988+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:49.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408280+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T20:09:08.408280+0000","last_peered":"2024-03-16T20:09:08.408280+0000","last_clean":"2024-03-16T20:09:08.408280+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:09:08.408280+0000","last_undegraded":"2024-03-16T20:09:08.408280+0000","last_fullsized":"2024-03-16T20:09:08.408280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647723+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T20:09:05.647723+0000","last_peered":"2024-03-16T20:09:05.647723+0000","last_clean":"2024-03-16T20:09:05.647723+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:09:05.647723+0000","last_undegraded":"2024-03-16T20:09:05.647723+0000","last_fullsized":"2024-03-16T20:09:05.647723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405802+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T20:09:09.405802+0000","last_peered":"2024-03-16T20:09:09.405802+0000","last_clean":"2024-03-16T20:09:09.405802+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:09:09.405802+0000","last_undegraded":"2024-03-16T20:09:09.405802+0000","last_fullsized":"2024-03-16T20:09:09.405802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476929,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476931,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476930,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476928,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509632,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509634,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542337,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:11:39.102 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-16T20:11:39.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:11:39.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 Processor -- start 2024-03-16T20:11:39.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 -- start start 2024-03-16T20:11:39.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c0c81e0 0x7f507c0c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f507c15b2a0 0x7f507c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c158c80 con 0x7f507c151ef0 2024-03-16T20:11:39.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0b0ad0 con 0x7f507c15b2a0 2024-03-16T20:11:39.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.280+0000 7f5081027640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c157cb0 con 0x7f507c0c81e0 2024-03-16T20:11:39.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.284+0000 7f507b59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f507c15b2a0 0x7f507c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.284+0000 7f507ad9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f507c151ef0 0x7f507c0c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51244/0 (socket says 172.21.15.32:51244) 2024-03-16T20:11:39.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.284+0000 7f507ad9d640 1 -- 172.21.15.32:0/2630428756 learned_addr learned my addr 172.21.15.32:0/2630428756 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:39.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.284+0000 7f507a59c640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c0c81e0 0x7f507c0c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f5079d9b640 1 -- 172.21.15.32:0/2630428756 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871904599 0 0) 0x7f507c158c80 con 0x7f507c151ef0 2024-03-16T20:11:39.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f5079d9b640 1 -- 172.21.15.32:0/2630428756 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5064003530 con 0x7f507c151ef0 2024-03-16T20:11:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f507b59e640 1 -- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c0c81e0 msgr2=0x7f507c0c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f507b59e640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c0c81e0 0x7f507c0c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f507b59e640 1 -- 172.21.15.32:0/2630428756 >> v1:172.21.15.32:6789/0 conn(0x7f507c151ef0 legacy=0x7f507c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.288+0000 7f507b59e640 1 -- 172.21.15.32:0/2630428756 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c15bfc0 con 0x7f507c15b2a0 2024-03-16T20:11:39.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.292+0000 7f507b59e640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c15b2a0 0x7f507c15b6d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5074009d90 tx=0x7f507401f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe60e42f1cca1b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5079d9b640 1 -- 172.21.15.32:0/2630428756 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50740af600 con 0x7f507c15b2a0 2024-03-16T20:11:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5079d9b640 1 -- 172.21.15.32:0/2630428756 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50740ab050 con 0x7f507c15b2a0 2024-03-16T20:11:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5079d9b640 1 -- 172.21.15.32:0/2630428756 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50740af980 con 0x7f507c15b2a0 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 -- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c15b2a0 msgr2=0x7f507c15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c15b2a0 0x7f507c15b6d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5074009d90 tx=0x7f507401f150 comp rx=0 tx=0).stop 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 -- 172.21.15.32:0/2630428756 shutdown_connections 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c15b2a0 0x7f507c15b6d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 --2- 172.21.15.32:0/2630428756 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c0c81e0 0x7f507c0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 -- 172.21.15.32:0/2630428756 >> 172.21.15.32:0/2630428756 conn(0x7f507c070f90 msgr2=0x7f507c0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 -- 172.21.15.32:0/2630428756 shutdown_connections 2024-03-16T20:11:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.300+0000 7f5081027640 1 -- 172.21.15.32:0/2630428756 wait complete. 2024-03-16T20:11:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 Processor -- start 2024-03-16T20:11:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 -- start start 2024-03-16T20:11:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c0c81e0 0x7f507c14d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c151ef0 0x7f507c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0b3390 con 0x7f507c15b2a0 2024-03-16T20:11:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c14de50 con 0x7f507c0c81e0 2024-03-16T20:11:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f5081027640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0b2a60 con 0x7f507c151ef0 2024-03-16T20:11:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f507b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f507ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c0c81e0 0x7f507c14d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f507a59c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c151ef0 0x7f507c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f507b59e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58056/0 (socket says 172.21.15.32:58056) 2024-03-16T20:11:39.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.304+0000 7f507b59e640 1 -- 172.21.15.32:0/745274575 learned_addr learned my addr 172.21.15.32:0/745274575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:39.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.308+0000 7f507b59e640 1 -- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c151ef0 msgr2=0x7f507c156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.308+0000 7f507b59e640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c151ef0 0x7f507c156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.308+0000 7f507b59e640 1 -- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c0c81e0 msgr2=0x7f507c14d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.308+0000 7f507b59e640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c0c81e0 0x7f507c14d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.308+0000 7f507b59e640 1 -- 172.21.15.32:0/745274575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074009a70 con 0x7f507c15b2a0 2024-03-16T20:11:39.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.312+0000 7f507b59e640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f50740b2040 tx=0x7f507401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:39.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.316+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074009450 con 0x7f507c15b2a0 2024-03-16T20:11:39.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.316+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50740b9410 con 0x7f507c15b2a0 2024-03-16T20:11:39.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.316+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50740b97c0 con 0x7f507c15b2a0 2024-03-16T20:11:39.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.320+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0b2d40 con 0x7f507c15b2a0 2024-03-16T20:11:39.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.320+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c14e130 con 0x7f507c15b2a0 2024-03-16T20:11:39.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.320+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c0b2ba0 con 0x7f507c15b2a0 2024-03-16T20:11:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.356+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f50740b6050 con 0x7f507c15b2a0 2024-03-16T20:11:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.356+0000 7f50637fe640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 0x7f5068069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.356+0000 7f507ad9d640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 0x7f5068069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.356+0000 7f507ad9d640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 0x7f5068069380 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f506c007fd0 tx=0x7f506c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.648+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f50740f41d0 con 0x7f507c15b2a0 2024-03-16T20:11:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.648+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f50740f0470 con 0x7f507c15b2a0 2024-03-16T20:11:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.816+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f507c0b31f0 con 0x7f5068066ec0 2024-03-16T20:11:39.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.824+0000 7f50637fe640 1 -- 172.21.15.32:0/745274575 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332622 (secure 0 0 0) 0x7f506c00f820 con 0x7f5068066ec0 2024-03-16T20:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:39.822 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:11:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 msgr2=0x7f5068069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 0x7f5068069380 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f506c007fd0 tx=0x7f506c009040 comp rx=0 tx=0).stop 2024-03-16T20:11:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 msgr2=0x7f507c14b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f50740b2040 tx=0x7f507401bef0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 shutdown_connections 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f5068066ec0 0x7f5068069380 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f507c15b2a0 0x7f507c14b0f0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f507c151ef0 0x7f507c156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 --2- 172.21.15.32:0/745274575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f507c0c81e0 0x7f507c14d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 >> 172.21.15.32:0/745274575 conn(0x7f507c070f90 msgr2=0x7f507c0c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 shutdown_connections 2024-03-16T20:11:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.828+0000 7f5081027640 1 -- 172.21.15.32:0/745274575 wait complete. 2024-03-16T20:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-03-16T20:11:38.846251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236864,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737856,"statfs":{"total":773094113280,"available":772851564544,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003550"},"pg_stats":[{"pgid":"2.1f","version":"32'14","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.407709+0000","last_change":"2024-03-16T19:56:33.963643+0000","last_active":"2024-03-16T20:09:03.407709+0000","last_peered":"2024-03-16T20:09:03.407709+0000","last_clean":"2024-03-16T20:09:03.407709+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:09:03.407709+0000","last_undegraded":"2024-03-16T20:09:03.407709+0000","last_fullsized":"2024-03-16T20:09:03.407709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.473025+0000","last_change":"2024-03-16T19:56:36.310250+0000","last_active":"2024-03-16T20:08:38.473025+0000","last_peered":"2024-03-16T20:08:38.473025+0000","last_clean":"2024-03-16T20:08:38.473025+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:08:38.473025+0000","last_undegraded":"2024-03-16T20:08:38.473025+0000","last_fullsized":"2024-03-16T20:08:38.473025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474069+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T20:08:48.474069+0000","last_peered":"2024-03-16T20:08:48.474069+0000","last_clean":"2024-03-16T20:08:48.474069+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:08:48.474069+0000","last_undegraded":"2024-03-16T20:08:48.474069+0000","last_fullsized":"2024-03-16T20:08:48.474069+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.472417+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T20:08:33.472417+0000","last_peered":"2024-03-16T20:08:33.472417+0000","last_clean":"2024-03-16T20:08:33.472417+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:08:33.472417+0000","last_undegraded":"2024-03-16T20:08:33.472417+0000","last_fullsized":"2024-03-16T20:08:33.472417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408303+0000","last_change":"2024-03-16T19:56:36.311083+0000","last_active":"2024-03-16T20:09:08.408303+0000","last_peered":"2024-03-16T20:09:08.408303+0000","last_clean":"2024-03-16T20:09:08.408303+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:09:08.408303+0000","last_undegraded":"2024-03-16T20:09:08.408303+0000","last_fullsized":"2024-03-16T20:09:08.408303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.479106+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T20:08:49.479106+0000","last_peered":"2024-03-16T20:08:49.479106+0000","last_clean":"2024-03-16T20:08:49.479106+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:08:49.479106+0000","last_undegraded":"2024-03-16T20:08:49.479106+0000","last_fullsized":"2024-03-16T20:08:49.479106+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'13","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405198+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T20:09:04.405198+0000","last_peered":"2024-03-16T20:09:04.405198+0000","last_clean":"2024-03-16T20:09:04.405198+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:09:04.405198+0000","last_undegraded":"2024-03-16T20:09:04.405198+0000","last_fullsized":"2024-03-16T20:09:04.405198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411271+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T20:06:39.411271+0000","last_peered":"2024-03-16T20:06:39.411271+0000","last_clean":"2024-03-16T20:06:39.411271+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:06:39.411271+0000","last_undegraded":"2024-03-16T20:06:39.411271+0000","last_fullsized":"2024-03-16T20:06:39.411271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464297+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T20:06:44.464297+0000","last_peered":"2024-03-16T20:06:44.464297+0000","last_clean":"2024-03-16T20:06:44.464297+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:06:44.464297+0000","last_undegraded":"2024-03-16T20:06:44.464297+0000","last_fullsized":"2024-03-16T20:06:44.464297+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405162+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T20:09:04.405162+0000","last_peered":"2024-03-16T20:09:04.405162+0000","last_clean":"2024-03-16T20:09:04.405162+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:09:04.405162+0000","last_undegraded":"2024-03-16T20:09:04.405162+0000","last_fullsized":"2024-03-16T20:09:04.405162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411290+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T20:06:39.411290+0000","last_peered":"2024-03-16T20:06:39.411290+0000","last_clean":"2024-03-16T20:06:39.411290+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:06:39.411290+0000","last_undegraded":"2024-03-16T20:06:39.411290+0000","last_fullsized":"2024-03-16T20:06:39.411290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:44.464225+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T20:06:44.464225+0000","last_peered":"2024-03-16T20:06:44.464225+0000","last_clean":"2024-03-16T20:06:44.464225+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:06:44.464225+0000","last_undegraded":"2024-03-16T20:06:44.464225+0000","last_fullsized":"2024-03-16T20:06:44.464225+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448622+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T20:07:29.448622+0000","last_peered":"2024-03-16T20:07:29.448622+0000","last_clean":"2024-03-16T20:07:29.448622+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:07:29.448622+0000","last_undegraded":"2024-03-16T20:07:29.448622+0000","last_fullsized":"2024-03-16T20:07:29.448622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.395875+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T20:07:49.395875+0000","last_peered":"2024-03-16T20:07:49.395875+0000","last_clean":"2024-03-16T20:07:49.395875+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:07:49.395875+0000","last_undegraded":"2024-03-16T20:07:49.395875+0000","last_fullsized":"2024-03-16T20:07:49.395875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473222+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T20:07:59.473222+0000","last_peered":"2024-03-16T20:07:59.473222+0000","last_clean":"2024-03-16T20:07:59.473222+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:07:59.473222+0000","last_undegraded":"2024-03-16T20:07:59.473222+0000","last_fullsized":"2024-03-16T20:07:59.473222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467453+0000","last_change":"2024-03-16T19:56:36.318718+0000","last_active":"2024-03-16T20:07:53.467453+0000","last_peered":"2024-03-16T20:07:53.467453+0000","last_clean":"2024-03-16T20:07:53.467453+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:07:53.467453+0000","last_undegraded":"2024-03-16T20:07:53.467453+0000","last_fullsized":"2024-03-16T20:07:53.467453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635909+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T20:07:25.635909+0000","last_peered":"2024-03-16T20:07:25.635909+0000","last_clean":"2024-03-16T20:07:25.635909+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:07:25.635909+0000","last_undegraded":"2024-03-16T20:07:25.635909+0000","last_fullsized":"2024-03-16T20:07:25.635909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:54.190915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473182+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T20:07:59.473182+0000","last_peered":"2024-03-16T20:07:59.473182+0000","last_clean":"2024-03-16T20:07:59.473182+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:07:59.473182+0000","last_undegraded":"2024-03-16T20:07:59.473182+0000","last_fullsized":"2024-03-16T20:07:59.473182+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463912+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T20:07:28.463912+0000","last_peered":"2024-03-16T20:07:28.463912+0000","last_clean":"2024-03-16T20:07:28.463912+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:07:28.463912+0000","last_undegraded":"2024-03-16T20:07:28.463912+0000","last_fullsized":"2024-03-16T20:07:28.463912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635877+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T20:07:25.635877+0000","last_peered":"2024-03-16T20:07:25.635877+0000","last_clean":"2024-03-16T20:07:25.635877+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:07:25.635877+0000","last_undegraded":"2024-03-16T20:07:25.635877+0000","last_fullsized":"2024-03-16T20:07:25.635877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:04.459381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:00.639712+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T20:08:00.639712+0000","last_peered":"2024-03-16T20:08:00.639712+0000","last_clean":"2024-03-16T20:08:00.639712+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:08:00.639712+0000","last_undegraded":"2024-03-16T20:08:00.639712+0000","last_fullsized":"2024-03-16T20:08:00.639712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:20.419750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.469295+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T20:07:29.469295+0000","last_peered":"2024-03-16T20:07:29.469295+0000","last_clean":"2024-03-16T20:07:29.469295+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:07:29.469295+0000","last_undegraded":"2024-03-16T20:07:29.469295+0000","last_fullsized":"2024-03-16T20:07:29.469295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.392918+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T20:07:24.392918+0000","last_peered":"2024-03-16T20:07:24.392918+0000","last_clean":"2024-03-16T20:07:24.392918+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:07:24.392918+0000","last_undegraded":"2024-03-16T20:07:24.392918+0000","last_fullsized":"2024-03-16T20:07:24.392918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473238+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T20:07:59.473238+0000","last_peered":"2024-03-16T20:07:59.473238+0000","last_clean":"2024-03-16T20:07:59.473238+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:07:59.473238+0000","last_undegraded":"2024-03-16T20:07:59.473238+0000","last_fullsized":"2024-03-16T20:07:59.473238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.395991+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T20:07:28.395991+0000","last_peered":"2024-03-16T20:07:28.395991+0000","last_clean":"2024-03-16T20:07:28.395991+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:07:28.395991+0000","last_undegraded":"2024-03-16T20:07:28.395991+0000","last_fullsized":"2024-03-16T20:07:28.395991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:53.429611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:24.448140+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T20:07:24.448140+0000","last_peered":"2024-03-16T20:07:24.448140+0000","last_clean":"2024-03-16T20:07:24.448140+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:07:24.448140+0000","last_undegraded":"2024-03-16T20:07:24.448140+0000","last_fullsized":"2024-03-16T20:07:24.448140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.481902+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T20:09:14.481902+0000","last_peered":"2024-03-16T20:09:14.481902+0000","last_clean":"2024-03-16T20:09:14.481902+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:09:14.481902+0000","last_undegraded":"2024-03-16T20:09:14.481902+0000","last_fullsized":"2024-03-16T20:09:14.481902+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:33.417264+0000","last_change":"2024-03-16T19:57:41.287408+0000","last_active":"2024-03-16T20:08:33.417264+0000","last_peered":"2024-03-16T20:08:33.417264+0000","last_clean":"2024-03-16T20:08:33.417264+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:08:33.417264+0000","last_undegraded":"2024-03-16T20:08:33.417264+0000","last_fullsized":"2024-03-16T20:08:33.417264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:54.981214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635961+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T20:07:25.635961+0000","last_peered":"2024-03-16T20:07:25.635961+0000","last_clean":"2024-03-16T20:07:25.635961+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:07:25.635961+0000","last_undegraded":"2024-03-16T20:07:25.635961+0000","last_fullsized":"2024-03-16T20:07:25.635961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:15.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473514+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T20:08:43.473514+0000","last_peered":"2024-03-16T20:08:43.473514+0000","last_clean":"2024-03-16T20:08:43.473514+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:08:43.473514+0000","last_undegraded":"2024-03-16T20:08:43.473514+0000","last_fullsized":"2024-03-16T20:08:43.473514+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647671+0000","last_change":"2024-03-16T19:56:36.235333+0000","last_active":"2024-03-16T20:09:05.647671+0000","last_peered":"2024-03-16T20:09:05.647671+0000","last_clean":"2024-03-16T20:09:05.647671+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:09:05.647671+0000","last_undegraded":"2024-03-16T20:09:05.647671+0000","last_fullsized":"2024-03-16T20:09:05.647671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:41:13.277162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405217+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T20:09:04.405217+0000","last_peered":"2024-03-16T20:09:04.405217+0000","last_clean":"2024-03-16T20:09:04.405217+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:09:04.405217+0000","last_undegraded":"2024-03-16T20:09:04.405217+0000","last_fullsized":"2024-03-16T20:09:04.405217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:13.408811+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T20:09:13.408811+0000","last_peered":"2024-03-16T20:09:13.408811+0000","last_clean":"2024-03-16T20:09:13.408811+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:09:13.408811+0000","last_undegraded":"2024-03-16T20:09:13.408811+0000","last_fullsized":"2024-03-16T20:09:13.408811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:30.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432177+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T20:09:09.432177+0000","last_peered":"2024-03-16T20:09:09.432177+0000","last_clean":"2024-03-16T20:09:09.432177+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:09:09.432177+0000","last_undegraded":"2024-03-16T20:09:09.432177+0000","last_fullsized":"2024-03-16T20:09:09.432177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647700+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T20:09:05.647700+0000","last_peered":"2024-03-16T20:09:05.647700+0000","last_clean":"2024-03-16T20:09:05.647700+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:09:05.647700+0000","last_undegraded":"2024-03-16T20:09:05.647700+0000","last_fullsized":"2024-03-16T20:09:05.647700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:33.482742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648736+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T20:09:15.648736+0000","last_peered":"2024-03-16T20:09:15.648736+0000","last_clean":"2024-03-16T20:09:15.648736+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:09:15.648736+0000","last_undegraded":"2024-03-16T20:09:15.648736+0000","last_fullsized":"2024-03-16T20:09:15.648736+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:16:59.054416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.481151+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T20:09:09.481151+0000","last_peered":"2024-03-16T20:09:09.481151+0000","last_clean":"2024-03-16T20:09:09.481151+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:09:09.481151+0000","last_undegraded":"2024-03-16T20:09:09.481151+0000","last_fullsized":"2024-03-16T20:09:09.481151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'6","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431117+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T20:09:04.431117+0000","last_peered":"2024-03-16T20:09:04.431117+0000","last_clean":"2024-03-16T20:09:04.431117+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:09:04.431117+0000","last_undegraded":"2024-03-16T20:09:04.431117+0000","last_fullsized":"2024-03-16T20:09:04.431117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:15.648771+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T20:09:15.648771+0000","last_peered":"2024-03-16T20:09:15.648771+0000","last_clean":"2024-03-16T20:09:15.648771+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:09:15.648771+0000","last_undegraded":"2024-03-16T20:09:15.648771+0000","last_fullsized":"2024-03-16T20:09:15.648771+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:19.183614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:35.643746+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T20:08:35.643746+0000","last_peered":"2024-03-16T20:08:35.643746+0000","last_clean":"2024-03-16T20:08:35.643746+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:08:35.643746+0000","last_undegraded":"2024-03-16T20:08:35.643746+0000","last_fullsized":"2024-03-16T20:08:35.643746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:54.205802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.420300+0000","last_change":"2024-03-16T19:56:33.900939+0000","last_active":"2024-03-16T20:09:03.420300+0000","last_peered":"2024-03-16T20:09:03.420300+0000","last_clean":"2024-03-16T20:09:03.420300+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:09:03.420300+0000","last_undegraded":"2024-03-16T20:09:03.420300+0000","last_fullsized":"2024-03-16T20:09:03.420300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:35.636979+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T20:07:35.636979+0000","last_peered":"2024-03-16T20:07:35.636979+0000","last_clean":"2024-03-16T20:07:35.636979+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:07:35.636979+0000","last_undegraded":"2024-03-16T20:07:35.636979+0000","last_fullsized":"2024-03-16T20:07:35.636979+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:19:30.334317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408207+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T20:09:08.408207+0000","last_peered":"2024-03-16T20:09:08.408207+0000","last_clean":"2024-03-16T20:09:08.408207+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:09:08.408207+0000","last_undegraded":"2024-03-16T20:09:08.408207+0000","last_fullsized":"2024-03-16T20:09:08.408207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:50.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:34.401181+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T20:08:34.401181+0000","last_peered":"2024-03-16T20:08:34.401181+0000","last_clean":"2024-03-16T20:08:34.401181+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:08:34.401181+0000","last_undegraded":"2024-03-16T20:08:34.401181+0000","last_fullsized":"2024-03-16T20:08:34.401181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:33.396597+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T20:07:33.396597+0000","last_peered":"2024-03-16T20:07:33.396597+0000","last_clean":"2024-03-16T20:07:33.396597+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:07:33.396597+0000","last_undegraded":"2024-03-16T20:07:33.396597+0000","last_fullsized":"2024-03-16T20:07:33.396597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.476309+0000","last_change":"2024-03-16T19:57:41.238831+0000","last_active":"2024-03-16T20:09:08.476309+0000","last_peered":"2024-03-16T20:09:08.476309+0000","last_clean":"2024-03-16T20:09:08.476309+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:09:08.476309+0000","last_undegraded":"2024-03-16T20:09:08.476309+0000","last_fullsized":"2024-03-16T20:09:08.476309+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:56:28.075437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431163+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T20:09:04.431163+0000","last_peered":"2024-03-16T20:09:04.431163+0000","last_clean":"2024-03-16T20:09:04.431163+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:09:04.431163+0000","last_undegraded":"2024-03-16T20:09:04.431163+0000","last_fullsized":"2024-03-16T20:09:04.431163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":197,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396571+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T20:07:54.396571+0000","last_peered":"2024-03-16T20:07:54.396571+0000","last_clean":"2024-03-16T20:07:54.396571+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:07:54.396571+0000","last_undegraded":"2024-03-16T20:07:54.396571+0000","last_fullsized":"2024-03-16T20:07:54.396571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461272+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T20:09:14.461272+0000","last_peered":"2024-03-16T20:09:14.461272+0000","last_clean":"2024-03-16T20:09:14.461272+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:09:14.461272+0000","last_undegraded":"2024-03-16T20:09:14.461272+0000","last_fullsized":"2024-03-16T20:09:14.461272+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396635+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T20:07:54.396635+0000","last_peered":"2024-03-16T20:07:54.396635+0000","last_clean":"2024-03-16T20:07:54.396635+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:07:54.396635+0000","last_undegraded":"2024-03-16T20:07:54.396635+0000","last_fullsized":"2024-03-16T20:07:54.396635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412513+0000","last_change":"2024-03-16T19:56:33.985067+0000","last_active":"2024-03-16T20:07:48.412513+0000","last_peered":"2024-03-16T20:07:48.412513+0000","last_clean":"2024-03-16T20:07:48.412513+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:07:48.412513+0000","last_undegraded":"2024-03-16T20:07:48.412513+0000","last_fullsized":"2024-03-16T20:07:48.412513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.449341+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T20:07:34.449341+0000","last_peered":"2024-03-16T20:07:34.449341+0000","last_clean":"2024-03-16T20:07:34.449341+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:07:34.449341+0000","last_undegraded":"2024-03-16T20:07:34.449341+0000","last_fullsized":"2024-03-16T20:07:34.449341+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421860+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T20:07:54.421860+0000","last_peered":"2024-03-16T20:07:54.421860+0000","last_clean":"2024-03-16T20:07:54.421860+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:07:54.421860+0000","last_undegraded":"2024-03-16T20:07:54.421860+0000","last_fullsized":"2024-03-16T20:07:54.421860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:50.638775+0000","last_change":"2024-03-16T19:56:33.857171+0000","last_active":"2024-03-16T20:07:50.638775+0000","last_peered":"2024-03-16T20:07:50.638775+0000","last_clean":"2024-03-16T20:07:50.638775+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:07:50.638775+0000","last_undegraded":"2024-03-16T20:07:50.638775+0000","last_fullsized":"2024-03-16T20:07:50.638775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:42.334279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:58.399907+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T20:07:58.399907+0000","last_peered":"2024-03-16T20:07:58.399907+0000","last_clean":"2024-03-16T20:07:58.399907+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:07:58.399907+0000","last_undegraded":"2024-03-16T20:07:58.399907+0000","last_fullsized":"2024-03-16T20:07:58.399907+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:20.488826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396733+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T20:07:54.396733+0000","last_peered":"2024-03-16T20:07:54.396733+0000","last_clean":"2024-03-16T20:07:54.396733+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:07:54.396733+0000","last_undegraded":"2024-03-16T20:07:54.396733+0000","last_fullsized":"2024-03-16T20:07:54.396733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:48.412548+0000","last_change":"2024-03-16T19:56:33.900440+0000","last_active":"2024-03-16T20:07:48.412548+0000","last_peered":"2024-03-16T20:07:48.412548+0000","last_clean":"2024-03-16T20:07:48.412548+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:07:48.412548+0000","last_undegraded":"2024-03-16T20:07:48.412548+0000","last_fullsized":"2024-03-16T20:07:48.412548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.419200+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T20:07:34.419200+0000","last_peered":"2024-03-16T20:07:34.419200+0000","last_clean":"2024-03-16T20:07:34.419200+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:07:34.419200+0000","last_undegraded":"2024-03-16T20:07:34.419200+0000","last_fullsized":"2024-03-16T20:07:34.419200+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396681+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T20:07:54.396681+0000","last_peered":"2024-03-16T20:07:54.396681+0000","last_clean":"2024-03-16T20:07:54.396681+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:07:54.396681+0000","last_undegraded":"2024-03-16T20:07:54.396681+0000","last_fullsized":"2024-03-16T20:07:54.396681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.471915+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T20:07:49.471915+0000","last_peered":"2024-03-16T20:07:49.471915+0000","last_clean":"2024-03-16T20:07:49.471915+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:07:49.471915+0000","last_undegraded":"2024-03-16T20:07:49.471915+0000","last_fullsized":"2024-03-16T20:07:49.471915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.469980+0000","last_change":"2024-03-16T19:57:41.242958+0000","last_active":"2024-03-16T20:07:34.469980+0000","last_peered":"2024-03-16T20:07:34.469980+0000","last_clean":"2024-03-16T20:07:34.469980+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:07:34.469980+0000","last_undegraded":"2024-03-16T20:07:34.469980+0000","last_fullsized":"2024-03-16T20:07:34.469980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:17:20.536712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399350+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T20:08:19.399350+0000","last_peered":"2024-03-16T20:08:19.399350+0000","last_clean":"2024-03-16T20:08:19.399350+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:08:19.399350+0000","last_undegraded":"2024-03-16T20:08:19.399350+0000","last_fullsized":"2024-03-16T20:08:19.399350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'18","reported_seq":60,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.415178+0000","last_change":"2024-03-16T19:56:33.900829+0000","last_active":"2024-03-16T20:08:13.415178+0000","last_peered":"2024-03-16T20:08:13.415178+0000","last_clean":"2024-03-16T20:08:13.415178+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:08:13.415178+0000","last_undegraded":"2024-03-16T20:08:13.415178+0000","last_fullsized":"2024-03-16T20:08:13.415178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.406062+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T20:08:48.406062+0000","last_peered":"2024-03-16T20:08:48.406062+0000","last_clean":"2024-03-16T20:08:48.406062+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:08:48.406062+0000","last_undegraded":"2024-03-16T20:08:48.406062+0000","last_fullsized":"2024-03-16T20:08:48.406062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:12.027347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.455260+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T20:08:19.455260+0000","last_peered":"2024-03-16T20:08:19.455260+0000","last_clean":"2024-03-16T20:08:19.455260+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:08:19.455260+0000","last_undegraded":"2024-03-16T20:08:19.455260+0000","last_fullsized":"2024-03-16T20:08:19.455260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'6","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454559+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T20:08:14.454559+0000","last_peered":"2024-03-16T20:08:14.454559+0000","last_clean":"2024-03-16T20:08:14.454559+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:08:14.454559+0000","last_undegraded":"2024-03-16T20:08:14.454559+0000","last_fullsized":"2024-03-16T20:08:14.454559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:48.474130+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T20:08:48.474130+0000","last_peered":"2024-03-16T20:08:48.474130+0000","last_clean":"2024-03-16T20:08:48.474130+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:08:48.474130+0000","last_undegraded":"2024-03-16T20:08:48.474130+0000","last_fullsized":"2024-03-16T20:08:48.474130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:11:12.582381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399380+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T20:08:19.399380+0000","last_peered":"2024-03-16T20:08:19.399380+0000","last_clean":"2024-03-16T20:08:19.399380+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:08:19.399380+0000","last_undegraded":"2024-03-16T20:08:19.399380+0000","last_fullsized":"2024-03-16T20:08:19.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.424455+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T20:08:14.424455+0000","last_peered":"2024-03-16T20:08:14.424455+0000","last_clean":"2024-03-16T20:08:14.424455+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:08:14.424455+0000","last_undegraded":"2024-03-16T20:08:14.424455+0000","last_fullsized":"2024-03-16T20:08:14.424455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470857+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T20:08:23.470857+0000","last_peered":"2024-03-16T20:08:23.470857+0000","last_clean":"2024-03-16T20:08:23.470857+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:08:23.470857+0000","last_undegraded":"2024-03-16T20:08:23.470857+0000","last_fullsized":"2024-03-16T20:08:23.470857+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:03.407915+0000","last_change":"2024-03-16T19:57:41.287279+0000","last_active":"2024-03-16T20:07:03.407915+0000","last_peered":"2024-03-16T20:07:03.407915+0000","last_clean":"2024-03-16T20:07:03.407915+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:07:03.407915+0000","last_undegraded":"2024-03-16T20:07:03.407915+0000","last_fullsized":"2024-03-16T20:07:03.407915+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:46.850016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:13.401433+0000","last_change":"2024-03-16T19:56:33.973534+0000","last_active":"2024-03-16T20:08:13.401433+0000","last_peered":"2024-03-16T20:08:13.401433+0000","last_clean":"2024-03-16T20:08:13.401433+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:08:13.401433+0000","last_undegraded":"2024-03-16T20:08:13.401433+0000","last_fullsized":"2024-03-16T20:08:13.401433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:36.771535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.429172+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T20:08:49.429172+0000","last_peered":"2024-03-16T20:08:49.429172+0000","last_clean":"2024-03-16T20:08:49.429172+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:08:49.429172+0000","last_undegraded":"2024-03-16T20:08:49.429172+0000","last_fullsized":"2024-03-16T20:08:49.429172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402419+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T20:08:44.402419+0000","last_peered":"2024-03-16T20:08:44.402419+0000","last_clean":"2024-03-16T20:08:44.402419+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:08:44.402419+0000","last_undegraded":"2024-03-16T20:08:44.402419+0000","last_fullsized":"2024-03-16T20:08:44.402419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454598+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T20:08:14.454598+0000","last_peered":"2024-03-16T20:08:14.454598+0000","last_clean":"2024-03-16T20:08:14.454598+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:08:14.454598+0000","last_undegraded":"2024-03-16T20:08:14.454598+0000","last_fullsized":"2024-03-16T20:08:14.454598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:25.642519+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T20:08:25.642519+0000","last_peered":"2024-03-16T20:08:25.642519+0000","last_clean":"2024-03-16T20:08:25.642519+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:08:25.642519+0000","last_undegraded":"2024-03-16T20:08:25.642519+0000","last_fullsized":"2024-03-16T20:08:25.642519+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:40:09.351583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.475987+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T20:08:19.475987+0000","last_peered":"2024-03-16T20:08:19.475987+0000","last_clean":"2024-03-16T20:08:19.475987+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:08:19.475987+0000","last_undegraded":"2024-03-16T20:08:19.475987+0000","last_fullsized":"2024-03-16T20:08:19.475987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644613+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T20:08:40.644613+0000","last_peered":"2024-03-16T20:08:40.644613+0000","last_clean":"2024-03-16T20:08:40.644613+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:08:40.644613+0000","last_undegraded":"2024-03-16T20:08:40.644613+0000","last_fullsized":"2024-03-16T20:08:40.644613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:03.782129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:24.476654+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T20:08:24.476654+0000","last_peered":"2024-03-16T20:08:24.476654+0000","last_clean":"2024-03-16T20:08:24.476654+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:08:24.476654+0000","last_undegraded":"2024-03-16T20:08:24.476654+0000","last_fullsized":"2024-03-16T20:08:24.476654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405275+0000","last_change":"2024-03-16T19:56:36.310914+0000","last_active":"2024-03-16T20:08:43.405275+0000","last_peered":"2024-03-16T20:08:43.405275+0000","last_clean":"2024-03-16T20:08:43.405275+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:08:43.405275+0000","last_undegraded":"2024-03-16T20:08:43.405275+0000","last_fullsized":"2024-03-16T20:08:43.405275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427645+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T20:08:39.427645+0000","last_peered":"2024-03-16T20:08:39.427645+0000","last_clean":"2024-03-16T20:08:39.427645+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:08:39.427645+0000","last_undegraded":"2024-03-16T20:08:39.427645+0000","last_fullsized":"2024-03-16T20:08:39.427645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470727+0000","last_change":"2024-03-16T19:57:41.246101+0000","last_active":"2024-03-16T20:08:23.470727+0000","last_peered":"2024-03-16T20:08:23.470727+0000","last_clean":"2024-03-16T20:08:23.470727+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:08:23.470727+0000","last_undegraded":"2024-03-16T20:08:23.470727+0000","last_fullsized":"2024-03-16T20:08:23.470727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:04.608965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.428389+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T20:08:44.428389+0000","last_peered":"2024-03-16T20:08:44.428389+0000","last_clean":"2024-03-16T20:08:44.428389+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:08:44.428389+0000","last_undegraded":"2024-03-16T20:08:44.428389+0000","last_fullsized":"2024-03-16T20:08:44.428389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.457446+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T20:08:39.457446+0000","last_peered":"2024-03-16T20:08:39.457446+0000","last_clean":"2024-03-16T20:08:39.457446+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:08:39.457446+0000","last_undegraded":"2024-03-16T20:08:39.457446+0000","last_fullsized":"2024-03-16T20:08:39.457446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:23.470791+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T20:08:23.470791+0000","last_peered":"2024-03-16T20:08:23.470791+0000","last_clean":"2024-03-16T20:08:23.470791+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:08:23.470791+0000","last_undegraded":"2024-03-16T20:08:23.470791+0000","last_fullsized":"2024-03-16T20:08:23.470791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:53.698453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.405199+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T20:08:43.405199+0000","last_peered":"2024-03-16T20:08:43.405199+0000","last_clean":"2024-03-16T20:08:43.405199+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:08:43.405199+0000","last_undegraded":"2024-03-16T20:08:43.405199+0000","last_fullsized":"2024-03-16T20:08:43.405199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:53.031515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:40.644545+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T20:08:40.644545+0000","last_peered":"2024-03-16T20:08:40.644545+0000","last_clean":"2024-03-16T20:08:40.644545+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:08:40.644545+0000","last_undegraded":"2024-03-16T20:08:40.644545+0000","last_fullsized":"2024-03-16T20:08:40.644545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:24.066974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427570+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T20:08:39.427570+0000","last_peered":"2024-03-16T20:08:39.427570+0000","last_clean":"2024-03-16T20:08:39.427570+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:08:39.427570+0000","last_undegraded":"2024-03-16T20:08:39.427570+0000","last_fullsized":"2024-03-16T20:08:39.427570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461319+0000","last_change":"2024-03-16T19:57:41.279710+0000","last_active":"2024-03-16T20:09:14.461319+0000","last_peered":"2024-03-16T20:09:14.461319+0000","last_clean":"2024-03-16T20:09:14.461319+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:09:14.461319+0000","last_undegraded":"2024-03-16T20:09:14.461319+0000","last_fullsized":"2024-03-16T20:09:14.461319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:02:54.725387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473548+0000","last_change":"2024-03-16T19:56:36.315085+0000","last_active":"2024-03-16T20:08:43.473548+0000","last_peered":"2024-03-16T20:08:43.473548+0000","last_clean":"2024-03-16T20:08:43.473548+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:08:43.473548+0000","last_undegraded":"2024-03-16T20:08:43.473548+0000","last_fullsized":"2024-03-16T20:08:43.473548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:38.472974+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T20:08:38.472974+0000","last_peered":"2024-03-16T20:08:38.472974+0000","last_clean":"2024-03-16T20:08:38.472974+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:08:38.472974+0000","last_undegraded":"2024-03-16T20:08:38.472974+0000","last_fullsized":"2024-03-16T20:08:38.472974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:20.574631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.459824+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T20:09:04.459824+0000","last_peered":"2024-03-16T20:09:04.459824+0000","last_clean":"2024-03-16T20:09:04.459824+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:09:04.459824+0000","last_undegraded":"2024-03-16T20:09:04.459824+0000","last_fullsized":"2024-03-16T20:09:04.459824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408266+0000","last_change":"2024-03-16T19:56:36.312638+0000","last_active":"2024-03-16T20:09:08.408266+0000","last_peered":"2024-03-16T20:09:08.408266+0000","last_clean":"2024-03-16T20:09:08.408266+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:09:08.408266+0000","last_undegraded":"2024-03-16T20:09:08.408266+0000","last_fullsized":"2024-03-16T20:09:08.408266+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.405682+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T20:06:43.405682+0000","last_peered":"2024-03-16T20:06:43.405682+0000","last_clean":"2024-03-16T20:06:43.405682+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:06:43.405682+0000","last_undegraded":"2024-03-16T20:06:43.405682+0000","last_fullsized":"2024-03-16T20:06:43.405682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:56.594618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:03.475556+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T20:09:03.475556+0000","last_peered":"2024-03-16T20:09:03.475556+0000","last_clean":"2024-03-16T20:09:03.475556+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:09:03.475556+0000","last_undegraded":"2024-03-16T20:09:03.475556+0000","last_fullsized":"2024-03-16T20:09:03.475556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:31.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.463549+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T20:06:39.463549+0000","last_peered":"2024-03-16T20:06:39.463549+0000","last_clean":"2024-03-16T20:06:39.463549+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:06:39.463549+0000","last_undegraded":"2024-03-16T20:06:39.463549+0000","last_fullsized":"2024-03-16T20:06:39.463549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:43.458436+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T20:06:43.458436+0000","last_peered":"2024-03-16T20:06:43.458436+0000","last_clean":"2024-03-16T20:06:43.458436+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:06:43.458436+0000","last_undegraded":"2024-03-16T20:06:43.458436+0000","last_fullsized":"2024-03-16T20:06:43.458436+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:48.778071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.460475+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T20:09:09.460475+0000","last_peered":"2024-03-16T20:09:09.460475+0000","last_clean":"2024-03-16T20:09:09.460475+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:09:09.460475+0000","last_undegraded":"2024-03-16T20:09:09.460475+0000","last_fullsized":"2024-03-16T20:09:09.460475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.420799+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T20:09:08.420799+0000","last_peered":"2024-03-16T20:09:08.420799+0000","last_clean":"2024-03-16T20:09:08.420799+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:09:08.420799+0000","last_undegraded":"2024-03-16T20:09:08.420799+0000","last_fullsized":"2024-03-16T20:09:08.420799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:16.393676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.431997+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T20:09:09.431997+0000","last_peered":"2024-03-16T20:09:09.431997+0000","last_clean":"2024-03-16T20:09:09.431997+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:09:09.431997+0000","last_undegraded":"2024-03-16T20:09:09.431997+0000","last_fullsized":"2024-03-16T20:09:09.431997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408250+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T20:09:08.408250+0000","last_peered":"2024-03-16T20:09:08.408250+0000","last_clean":"2024-03-16T20:09:08.408250+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:09:08.408250+0000","last_undegraded":"2024-03-16T20:09:08.408250+0000","last_fullsized":"2024-03-16T20:09:08.408250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.463977+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T20:07:28.463977+0000","last_peered":"2024-03-16T20:07:28.463977+0000","last_clean":"2024-03-16T20:07:28.463977+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:07:28.463977+0000","last_undegraded":"2024-03-16T20:07:28.463977+0000","last_fullsized":"2024-03-16T20:07:28.463977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:25.635930+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T20:07:25.635930+0000","last_peered":"2024-03-16T20:07:25.635930+0000","last_clean":"2024-03-16T20:07:25.635930+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:07:25.635930+0000","last_undegraded":"2024-03-16T20:07:25.635930+0000","last_fullsized":"2024-03-16T20:07:25.635930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:31:21.727819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448666+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T20:07:29.448666+0000","last_peered":"2024-03-16T20:07:29.448666+0000","last_clean":"2024-03-16T20:07:29.448666+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:07:29.448666+0000","last_undegraded":"2024-03-16T20:07:29.448666+0000","last_fullsized":"2024-03-16T20:07:29.448666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:28.464010+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T20:07:28.464010+0000","last_peered":"2024-03-16T20:07:28.464010+0000","last_clean":"2024-03-16T20:07:28.464010+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:07:28.464010+0000","last_undegraded":"2024-03-16T20:07:28.464010+0000","last_fullsized":"2024-03-16T20:07:28.464010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.418412+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T20:07:29.418412+0000","last_peered":"2024-03-16T20:07:29.418412+0000","last_clean":"2024-03-16T20:07:29.418412+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:07:29.418412+0000","last_undegraded":"2024-03-16T20:07:29.418412+0000","last_fullsized":"2024-03-16T20:07:29.418412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.467410+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T20:07:53.467410+0000","last_peered":"2024-03-16T20:07:53.467410+0000","last_clean":"2024-03-16T20:07:53.467410+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:07:53.467410+0000","last_undegraded":"2024-03-16T20:07:53.467410+0000","last_fullsized":"2024-03-16T20:07:53.467410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421807+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T20:07:54.421807+0000","last_peered":"2024-03-16T20:07:54.421807+0000","last_clean":"2024-03-16T20:07:54.421807+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:07:54.421807+0000","last_undegraded":"2024-03-16T20:07:54.421807+0000","last_fullsized":"2024-03-16T20:07:54.421807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:53.399380+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T20:07:53.399380+0000","last_peered":"2024-03-16T20:07:53.399380+0000","last_clean":"2024-03-16T20:07:53.399380+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:07:53.399380+0000","last_undegraded":"2024-03-16T20:07:53.399380+0000","last_fullsized":"2024-03-16T20:07:53.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.452068+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T20:07:54.452068+0000","last_peered":"2024-03-16T20:07:54.452068+0000","last_clean":"2024-03-16T20:07:54.452068+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:07:54.452068+0000","last_undegraded":"2024-03-16T20:07:54.452068+0000","last_fullsized":"2024-03-16T20:07:54.452068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396659+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T20:07:54.396659+0000","last_peered":"2024-03-16T20:07:54.396659+0000","last_clean":"2024-03-16T20:07:54.396659+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:07:54.396659+0000","last_undegraded":"2024-03-16T20:07:54.396659+0000","last_fullsized":"2024-03-16T20:07:54.396659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.415607+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T20:08:18.415607+0000","last_peered":"2024-03-16T20:08:18.415607+0000","last_clean":"2024-03-16T20:08:18.415607+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:08:18.415607+0000","last_undegraded":"2024-03-16T20:08:18.415607+0000","last_fullsized":"2024-03-16T20:08:18.415607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425243+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T20:08:19.425243+0000","last_peered":"2024-03-16T20:08:19.425243+0000","last_clean":"2024-03-16T20:08:19.425243+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:08:19.425243+0000","last_undegraded":"2024-03-16T20:08:19.425243+0000","last_fullsized":"2024-03-16T20:08:19.425243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425164+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T20:08:19.425164+0000","last_peered":"2024-03-16T20:08:19.425164+0000","last_clean":"2024-03-16T20:08:19.425164+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:08:19.425164+0000","last_undegraded":"2024-03-16T20:08:19.425164+0000","last_fullsized":"2024-03-16T20:08:19.425164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:15.641445+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T20:08:15.641445+0000","last_peered":"2024-03-16T20:08:15.641445+0000","last_clean":"2024-03-16T20:08:15.641445+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:08:15.641445+0000","last_undegraded":"2024-03-16T20:08:15.641445+0000","last_fullsized":"2024-03-16T20:08:15.641445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:02.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:18.401956+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T20:08:18.401956+0000","last_peered":"2024-03-16T20:08:18.401956+0000","last_clean":"2024-03-16T20:08:18.401956+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:08:18.401956+0000","last_undegraded":"2024-03-16T20:08:18.401956+0000","last_fullsized":"2024-03-16T20:08:18.401956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:29.802188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:43.473580+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T20:08:43.473580+0000","last_peered":"2024-03-16T20:08:43.473580+0000","last_clean":"2024-03-16T20:08:43.473580+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:08:43.473580+0000","last_undegraded":"2024-03-16T20:08:43.473580+0000","last_fullsized":"2024-03-16T20:08:43.473580+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.478645+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T20:08:44.478645+0000","last_peered":"2024-03-16T20:08:44.478645+0000","last_clean":"2024-03-16T20:08:44.478645+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:08:44.478645+0000","last_undegraded":"2024-03-16T20:08:44.478645+0000","last_fullsized":"2024-03-16T20:08:44.478645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457937+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T20:08:44.457937+0000","last_peered":"2024-03-16T20:08:44.457937+0000","last_clean":"2024-03-16T20:08:44.457937+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:08:44.457937+0000","last_undegraded":"2024-03-16T20:08:44.457937+0000","last_fullsized":"2024-03-16T20:08:44.457937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457968+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T20:08:44.457968+0000","last_peered":"2024-03-16T20:08:44.457968+0000","last_clean":"2024-03-16T20:08:44.457968+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:08:44.457968+0000","last_undegraded":"2024-03-16T20:08:44.457968+0000","last_fullsized":"2024-03-16T20:08:44.457968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402472+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T20:08:44.402472+0000","last_peered":"2024-03-16T20:08:44.402472+0000","last_clean":"2024-03-16T20:08:44.402472+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:08:44.402472+0000","last_undegraded":"2024-03-16T20:08:44.402472+0000","last_fullsized":"2024-03-16T20:08:44.402472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.476255+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T20:09:08.476255+0000","last_peered":"2024-03-16T20:09:08.476255+0000","last_clean":"2024-03-16T20:09:08.476255+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:09:08.476255+0000","last_undegraded":"2024-03-16T20:09:08.476255+0000","last_fullsized":"2024-03-16T20:09:08.476255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:43.434379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387104+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T20:06:39.387104+0000","last_peered":"2024-03-16T20:06:39.387104+0000","last_clean":"2024-03-16T20:06:39.387104+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:06:39.387104+0000","last_undegraded":"2024-03-16T20:06:39.387104+0000","last_fullsized":"2024-03-16T20:06:39.387104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.411230+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T20:06:39.411230+0000","last_peered":"2024-03-16T20:06:39.411230+0000","last_clean":"2024-03-16T20:06:39.411230+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:06:39.411230+0000","last_undegraded":"2024-03-16T20:06:39.411230+0000","last_fullsized":"2024-03-16T20:06:39.411230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:06:39.387143+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T20:06:39.387143+0000","last_peered":"2024-03-16T20:06:39.387143+0000","last_clean":"2024-03-16T20:06:39.387143+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:06:39.387143+0000","last_undegraded":"2024-03-16T20:06:39.387143+0000","last_fullsized":"2024-03-16T20:06:39.387143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:43.411988+0000","last_change":"2024-03-16T19:57:41.236414+0000","last_active":"2024-03-16T20:07:43.411988+0000","last_peered":"2024-03-16T20:07:43.411988+0000","last_clean":"2024-03-16T20:07:43.411988+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:07:43.411988+0000","last_undegraded":"2024-03-16T20:07:43.411988+0000","last_fullsized":"2024-03-16T20:07:43.411988+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:49.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:08.408280+0000","last_change":"2024-03-16T19:56:36.305396+0000","last_active":"2024-03-16T20:09:08.408280+0000","last_peered":"2024-03-16T20:09:08.408280+0000","last_clean":"2024-03-16T20:09:08.408280+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:09:08.408280+0000","last_undegraded":"2024-03-16T20:09:08.408280+0000","last_fullsized":"2024-03-16T20:09:08.408280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:10.040658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:05.647723+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T20:09:05.647723+0000","last_peered":"2024-03-16T20:09:05.647723+0000","last_clean":"2024-03-16T20:09:05.647723+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:09:05.647723+0000","last_undegraded":"2024-03-16T20:09:05.647723+0000","last_fullsized":"2024-03-16T20:09:05.647723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:11.975333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405802+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T20:09:09.405802+0000","last_peered":"2024-03-16T20:09:09.405802+0000","last_clean":"2024-03-16T20:09:09.405802+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:09:09.405802+0000","last_undegraded":"2024-03-16T20:09:09.405802+0000","last_fullsized":"2024-03-16T20:09:09.405802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476929,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476931,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476930,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476928,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509632,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509634,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542337,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:11:39.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:11:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 Processor -- start 2024-03-16T20:11:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 -- start start 2024-03-16T20:11:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d00725e0 0x7f20d0073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f20d015a420 0x7f20d015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d0151020 con 0x7f20d0150c40 2024-03-16T20:11:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d015a000 con 0x7f20d015a420 2024-03-16T20:11:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:39.996+0000 7f20d4d1b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015adf0 con 0x7f20d00725e0 2024-03-16T20:11:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.000+0000 7f20cdd9b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d00725e0 0x7f20d0073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.000+0000 7f20ced9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f20d015a420 0x7f20d015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.000+0000 7f20ced9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f20d015a420 0x7f20d015a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:58544/0 (socket says 172.21.15.32:58544) 2024-03-16T20:11:39.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.000+0000 7f20ced9d640 1 -- 172.21.15.32:0/1125735533 learned_addr learned my addr 172.21.15.32:0/1125735533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.004+0000 7f20ced9d640 1 -- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d00725e0 msgr2=0x7f20d0073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.004+0000 7f20ced9d640 1 --2- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d00725e0 0x7f20d0073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.004+0000 7f20ced9d640 1 -- 172.21.15.32:0/1125735533 >> v1:172.21.15.59:6789/0 conn(0x7f20d0150c40 legacy=0x7f20d0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.004+0000 7f20ced9d640 1 -- 172.21.15.32:0/1125735533 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d0072da0 con 0x7f20d015a420 2024-03-16T20:11:40.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.012+0000 7f20ced9d640 1 --2- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d015a420 0x7f20d015a8b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f20d00c4a30 tx=0x7f20c801c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8040a5462779795 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.020+0000 7f20cd59a640 1 -- 172.21.15.32:0/1125735533 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c80a5850 con 0x7f20d015a420 2024-03-16T20:11:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.020+0000 7f20cd59a640 1 -- 172.21.15.32:0/1125735533 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c80a59b0 con 0x7f20d015a420 2024-03-16T20:11:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.020+0000 7f20cd59a640 1 -- 172.21.15.32:0/1125735533 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c80a5d40 con 0x7f20d015a420 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.020+0000 7f20d4d1b640 1 -- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d015a420 msgr2=0x7f20d015a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.020+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d015a420 0x7f20d015a8b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f20d00c4a30 tx=0x7f20c801c9d0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- 172.21.15.32:0/1125735533 shutdown_connections 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d015a420 0x7f20d015a8b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/1125735533 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d00725e0 0x7f20d0073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- 172.21.15.32:0/1125735533 >> 172.21.15.32:0/1125735533 conn(0x7f20d006c0d0 msgr2=0x7f20d006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- 172.21.15.32:0/1125735533 shutdown_connections 2024-03-16T20:11:40.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- 172.21.15.32:0/1125735533 wait complete. 2024-03-16T20:11:40.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 Processor -- start 2024-03-16T20:11:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- start start 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d00725e0 0x7f20d014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d015a420 0x7f20d00cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00ca4d0 con 0x7f20d0150c40 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d014d410 con 0x7f20d00725e0 2024-03-16T20:11:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.024+0000 7f20d4d1b640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00ca720 con 0x7f20d015a420 2024-03-16T20:11:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.028+0000 7f20cdd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.028+0000 7f20ced9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d015a420 0x7f20d00cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.028+0000 7f20cdd9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58058/0 (socket says 172.21.15.32:58058) 2024-03-16T20:11:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.028+0000 7f20cdd9b640 1 -- 172.21.15.32:0/2831273712 learned_addr learned my addr 172.21.15.32:0/2831273712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:40.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.028+0000 7f20ce59c640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d00725e0 0x7f20d014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.032+0000 7f20cdd9b640 1 -- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d015a420 msgr2=0x7f20d00cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.032+0000 7f20cdd9b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d015a420 0x7f20d00cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.032+0000 7f20cdd9b640 1 -- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d00725e0 msgr2=0x7f20d014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.032+0000 7f20cdd9b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d00725e0 0x7f20d014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.032+0000 7f20cdd9b640 1 -- 172.21.15.32:0/2831273712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c8009b30 con 0x7f20d0150c40 2024-03-16T20:11:40.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.036+0000 7f20cdd9b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e870 tx=0x7f20c400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4009510 con 0x7f20d0150c40 2024-03-16T20:11:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c400eec0 con 0x7f20d0150c40 2024-03-16T20:11:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d00caa00 con 0x7f20d0150c40 2024-03-16T20:11:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c400cb40 con 0x7f20d0150c40 2024-03-16T20:11:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d014be30 con 0x7f20d0150c40 2024-03-16T20:11:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.040+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d0068000 con 0x7f20d0150c40 2024-03-16T20:11:40.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.320+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f20c400cca0 con 0x7f20d0150c40 2024-03-16T20:11:40.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.320+0000 7f20beffd640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 0x7f20b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:40.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.320+0000 7f20ce59c640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 0x7f20b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.324+0000 7f20ce59c640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 0x7f20b0069300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007fd0 tx=0x7f20c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:40.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.348+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f20c4055ed0 con 0x7f20d0150c40 2024-03-16T20:11:40.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.348+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f20c4052170 con 0x7f20d0150c40 2024-03-16T20:11:40.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.608+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20d00c7960 con 0x7f20d0150c40 2024-03-16T20:11:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.660+0000 7f20beffd640 1 -- 172.21.15.32:0/2831273712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+18971 (secure 0 0 0) 0x7f20c4017090 con 0x7f20d0150c40 2024-03-16T20:11:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:40.653 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"fsid":"fb85817b-8df0-4e27-a0bc-dc277b22543a","created":"2024-03-16T19:55:30.844341+0000","modified":"2024-03-16T19:58:20.305611+0000","last_up_change":"2024-03-16T19:55:53.242247+0000","last_in_change":"2024-03-16T19:55:42.235009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:55:53.065249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:56:31.637719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:56:33.839961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T19:56:40.663264+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2b55a720-9297-4595-89bb-0e9a1f3da471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6811","nonce":1875165640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6813","nonce":1875165640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6817","nonce":1875165640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1875165640},{"type":"v1","addr":"172.21.15.32:6815","nonce":1875165640}]},"public_addr":"172.21.15.32:6811/1875165640","cluster_addr":"172.21.15.32:6813/1875165640","heartbeat_back_addr":"172.21.15.32:6817/1875165640","heartbeat_front_addr":"172.21.15.32:6815/1875165640","state":["exists","up"]},{"osd":1,"uuid":"30704413-befa-4817-92b0-d0f96b5d354d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6803","nonce":1994708459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6805","nonce":1994708459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6809","nonce":1994708459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1994708459},{"type":"v1","addr":"172.21.15.32:6807","nonce":1994708459}]},"public_addr":"172.21.15.32:6803/1994708459","cluster_addr":"172.21.15.32:6805/1994708459","heartbeat_back_addr":"172.21.15.32:6809/1994708459","heartbeat_front_addr":"172.21.15.32:6807/1994708459","state":["exists","up"]},{"osd":2,"uuid":"18500384-71d8-40a4-a508-b47407b9d67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6819","nonce":3435819214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6821","nonce":3435819214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6825","nonce":3435819214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3435819214},{"type":"v1","addr":"172.21.15.32:6823","nonce":3435819214}]},"public_addr":"172.21.15.32:6819/3435819214","cluster_addr":"172.21.15.32:6821/3435819214","heartbeat_back_addr":"172.21.15.32:6825/3435819214","heartbeat_front_addr":"172.21.15.32:6823/3435819214","state":["exists","up"]},{"osd":3,"uuid":"b04758b0-8a06-46e4-8061-232f252de437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6827","nonce":1974622921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6829","nonce":1974622921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6833","nonce":1974622921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1974622921},{"type":"v1","addr":"172.21.15.32:6831","nonce":1974622921}]},"public_addr":"172.21.15.32:6827/1974622921","cluster_addr":"172.21.15.32:6829/1974622921","heartbeat_back_addr":"172.21.15.32:6833/1974622921","heartbeat_front_addr":"172.21.15.32:6831/1974622921","state":["exists","up"]},{"osd":4,"uuid":"3358c3b9-7e70-42e6-b285-3865a95d8b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6825","nonce":446247080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6827","nonce":446247080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6831","nonce":446247080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":446247080},{"type":"v1","addr":"172.21.15.59:6829","nonce":446247080}]},"public_addr":"172.21.15.59:6825/446247080","cluster_addr":"172.21.15.59:6827/446247080","heartbeat_back_addr":"172.21.15.59:6831/446247080","heartbeat_front_addr":"172.21.15.59:6829/446247080","state":["exists","up"]},{"osd":5,"uuid":"28be6473-1ce4-448f-bfa9-437370b18219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6817","nonce":2401239785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6819","nonce":2401239785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6823","nonce":2401239785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2401239785},{"type":"v1","addr":"172.21.15.59:6821","nonce":2401239785}]},"public_addr":"172.21.15.59:6817/2401239785","cluster_addr":"172.21.15.59:6819/2401239785","heartbeat_back_addr":"172.21.15.59:6823/2401239785","heartbeat_front_addr":"172.21.15.59:6821/2401239785","state":["exists","up"]},{"osd":6,"uuid":"22cff389-1c2e-4b39-914c-f06fcf60a0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6803","nonce":459431804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6807","nonce":459431804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6815","nonce":459431804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":459431804},{"type":"v1","addr":"172.21.15.59:6811","nonce":459431804}]},"public_addr":"172.21.15.59:6803/459431804","cluster_addr":"172.21.15.59:6807/459431804","heartbeat_back_addr":"172.21.15.59:6815/459431804","heartbeat_front_addr":"172.21.15.59:6811/459431804","state":["exists","up"]},{"osd":7,"uuid":"4c2d369f-f6e2-4644-b752-e10b0d42117b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6802","nonce":1528567805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6806","nonce":1528567805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6814","nonce":1528567805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1528567805},{"type":"v1","addr":"172.21.15.59:6810","nonce":1528567805}]},"public_addr":"172.21.15.59:6802/1528567805","cluster_addr":"172.21.15.59:6806/1528567805","heartbeat_back_addr":"172.21.15.59:6814/1528567805","heartbeat_front_addr":"172.21.15.59:6810/1528567805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.081430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:47.782777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.088487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.153137+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.141094+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.205865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.053855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:55:48.090535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6835/1407783290":"2024-03-17T19:58:20.269769+0000","172.21.15.59:6834/1407783290":"2024-03-17T19:58:20.269769+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:11:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 msgr2=0x7f20b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 0x7f20b0069300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007fd0 tx=0x7f20c0009040 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 msgr2=0x7f20d01475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e870 tx=0x7f20c400ec70 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 shutdown_connections 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f20b0066e40 0x7f20b0069300 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f20d015a420 0x7f20d00cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20d0150c40 0x7f20d01475b0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 --2- 172.21.15.32:0/2831273712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20d00725e0 0x7f20d014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 >> 172.21.15.32:0/2831273712 conn(0x7f20d006c0d0 msgr2=0x7f20d014e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 shutdown_connections 2024-03-16T20:11:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:40.664+0000 7f20d4d1b640 1 -- 172.21.15.32:0/2831273712 wait complete. 2024-03-16T20:11:41.673 INFO:tasks.ceph:Scrubbing osd.0 2024-03-16T20:11:41.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.792+0000 7f9042ff6640 1 Processor -- start 2024-03-16T20:11:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.792+0000 7f9042ff6640 1 -- start start 2024-03-16T20:11:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9042ff6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9042ff6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f903c06fac0 0x7f903c162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9042ff6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c0c88d0 con 0x7f903c1521e0 2024-03-16T20:11:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9042ff6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c06f6a0 con 0x7f903c06fac0 2024-03-16T20:11:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9042ff6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c3360 con 0x7f903c151e00 2024-03-16T20:11:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f903bfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f903c1521e0 0x7f903c0cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51258/0 (socket says 172.21.15.32:51258) 2024-03-16T20:11:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f903bfff640 1 -- 172.21.15.32:0/313230083 learned_addr learned my addr 172.21.15.32:0/313230083 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9040d92640 1 --2- 172.21.15.32:0/313230083 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.796+0000 7f9041593640 1 --2- 172.21.15.32:0/313230083 >> v2:172.21.15.59:3300/0 conn(0x7f903c06fac0 0x7f903c162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f903b7fe640 1 -- 172.21.15.32:0/313230083 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262266043 0 0) 0x7f903c0c88d0 con 0x7f903c1521e0 2024-03-16T20:11:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f903b7fe640 1 -- 172.21.15.32:0/313230083 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c003530 con 0x7f903c1521e0 2024-03-16T20:11:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f9040d92640 1 -- 172.21.15.32:0/313230083 >> v2:172.21.15.59:3300/0 conn(0x7f903c06fac0 msgr2=0x7f903c162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f9040d92640 1 --2- 172.21.15.32:0/313230083 >> v2:172.21.15.59:3300/0 conn(0x7f903c06fac0 0x7f903c162590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f9040d92640 1 -- 172.21.15.32:0/313230083 >> v1:172.21.15.32:6789/0 conn(0x7f903c1521e0 legacy=0x7f903c0cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.800+0000 7f9040d92640 1 -- 172.21.15.32:0/313230083 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c157c70 con 0x7f903c151e00 2024-03-16T20:11:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.804+0000 7f9040d92640 1 --2- 172.21.15.32:0/313230083 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c0ca810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90300097c0 tx=0x7f903001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7d20dd1a005a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.812+0000 7f903b7fe640 1 -- 172.21.15.32:0/313230083 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90300ab600 con 0x7f903c151e00 2024-03-16T20:11:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.812+0000 7f903b7fe640 1 -- 172.21.15.32:0/313230083 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90300a7050 con 0x7f903c151e00 2024-03-16T20:11:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.812+0000 7f903b7fe640 1 -- 172.21.15.32:0/313230083 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90300ab960 con 0x7f903c151e00 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- 172.21.15.32:0/313230083 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 msgr2=0x7f903c0ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- 172.21.15.32:0/313230083 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c0ca810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90300097c0 tx=0x7f903001c4c0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- 172.21.15.32:0/313230083 shutdown_connections 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- 172.21.15.32:0/313230083 >> v2:172.21.15.59:3300/0 conn(0x7f903c06fac0 0x7f903c162590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- 172.21.15.32:0/313230083 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c0ca810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- 172.21.15.32:0/313230083 >> 172.21.15.32:0/313230083 conn(0x7f903c06b5c0 msgr2=0x7f903c066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- 172.21.15.32:0/313230083 shutdown_connections 2024-03-16T20:11:41.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- 172.21.15.32:0/313230083 wait complete. 2024-03-16T20:11:41.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 Processor -- start 2024-03-16T20:11:41.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- start start 2024-03-16T20:11:41.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 0x7f903c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c1521e0 0x7f903c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c161b00 con 0x7f903c06fac0 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c154ef0 con 0x7f903c1521e0 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9042ff6640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c160ea0 con 0x7f903c151e00 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9040d92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 0x7f903c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9041593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c1521e0 0x7f903c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9040d92640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 0x7f903c1549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58064/0 (socket says 172.21.15.32:58064) 2024-03-16T20:11:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.816+0000 7f9040d92640 1 -- 172.21.15.32:0/2165448186 learned_addr learned my addr 172.21.15.32:0/2165448186 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:41.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.820+0000 7f903bfff640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.824+0000 7f9040d92640 1 -- 172.21.15.32:0/2165448186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 msgr2=0x7f903c070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.824+0000 7f9040d92640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f903c151e00 0x7f903c070f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.824+0000 7f9040d92640 1 -- 172.21.15.32:0/2165448186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c1521e0 msgr2=0x7f903c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.824+0000 7f9040d92640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c1521e0 0x7f903c06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.824+0000 7f9040d92640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030009420 con 0x7f903c06fac0 2024-03-16T20:11:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.828+0000 7f9040d92640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 0x7f903c1549b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f90300ae040 tx=0x7f9030019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030003e60 con 0x7f903c06fac0 2024-03-16T20:11:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9030004520 con 0x7f903c06fac0 2024-03-16T20:11:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c1f6fb0 con 0x7f903c06fac0 2024-03-16T20:11:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90300b4630 con 0x7f903c06fac0 2024-03-16T20:11:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c161440 con 0x7f903c06fac0 2024-03-16T20:11:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.832+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f903c0c3620 con 0x7f903c06fac0 2024-03-16T20:11:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.864+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f90300b2040 con 0x7f903c06fac0 2024-03-16T20:11:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.864+0000 7f90397fa640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9020066ec0 0x7f9020069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.864+0000 7f903bfff640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9020066ec0 0x7f9020069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.864+0000 7f903bfff640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9020066ec0 0x7f9020069380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9028007fd0 tx=0x7f9028009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:41.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.868+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f90300eff70 con 0x7f903c06fac0 2024-03-16T20:11:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.868+0000 7f90397fa640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7f9020070960 0x7f9020072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.868+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90200734d0 con 0x7f9020070960 2024-03-16T20:11:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.868+0000 7f9041593640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7f9020070960 0x7f9020072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.868+0000 7f9041593640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7f9020070960 0x7f9020072e00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.872+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== osd.0 v2:172.21.15.32:6810/1875165640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f90200734d0 con 0x7f9020070960 2024-03-16T20:11:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.872+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90300ec210 con 0x7f903c06fac0 2024-03-16T20:11:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 --> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f903c0cad50 con 0x7f9020070960 2024-03-16T20:11:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f90397fa640 1 -- 172.21.15.32:0/2165448186 <== osd.0 v2:172.21.15.32:6810/1875165640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f903400f040 con 0x7f9020070960 2024-03-16T20:11:41.889 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:41.890 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:41.890 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7f9020070960 msgr2=0x7f9020072e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6810/1875165640,v1:172.21.15.32:6811/1875165640] conn(0x7f9020070960 0x7f9020072e00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9020066ec0 msgr2=0x7f9020069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9020066ec0 0x7f9020069380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9028007fd0 tx=0x7f9028009040 comp rx=0 tx=0).stop 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 msgr2=0x7f903c1549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 --2- 172.21.15.32:0/2165448186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f903c06fac0 0x7f903c1549b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f90300ae040 tx=0x7f9030019260 comp rx=0 tx=0).stop 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9041593640 1 -- 172.21.15.32:0/2165448186 reap_dead start 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 shutdown_connections 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 >> 172.21.15.32:0/2165448186 conn(0x7f903c06b5c0 msgr2=0x7f903c0cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 shutdown_connections 2024-03-16T20:11:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:41.896+0000 7f9042ff6640 1 -- 172.21.15.32:0/2165448186 wait complete. 2024-03-16T20:11:41.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-16T20:11:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 Processor -- start 2024-03-16T20:11:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 -- start start 2024-03-16T20:11:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbce81542e0 0x7fbce8070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce815a1d0 0x7fbce815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce81546c0 con 0x7fbce8073090 2024-03-16T20:11:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce815d4b0 con 0x7fbce81542e0 2024-03-16T20:11:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.024+0000 7fbced761640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce815d6c0 con 0x7fbce815a1d0 2024-03-16T20:11:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.028+0000 7fbce67fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbce8073090 0x7fbce8072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51268/0 (socket says 172.21.15.32:51268) 2024-03-16T20:11:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.028+0000 7fbce67fc640 1 -- 172.21.15.32:0/3797038373 learned_addr learned my addr 172.21.15.32:0/3797038373 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.028+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/3797038373 >> v2:172.21.15.59:3300/0 conn(0x7fbce81542e0 0x7fbce8070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.028+0000 7fbce77fe640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce815a1d0 0x7fbce815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce5ffb640 1 -- 172.21.15.32:0/3797038373 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678042174 0 0) 0x7fbce81546c0 con 0x7fbce8073090 2024-03-16T20:11:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce5ffb640 1 -- 172.21.15.32:0/3797038373 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd0003530 con 0x7fbce8073090 2024-03-16T20:11:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce6ffd640 1 -- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce815a1d0 msgr2=0x7fbce815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce815a1d0 0x7fbce815a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce6ffd640 1 -- 172.21.15.32:0/3797038373 >> v1:172.21.15.32:6789/0 conn(0x7fbce8073090 legacy=0x7fbce8072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.032+0000 7fbce6ffd640 1 -- 172.21.15.32:0/3797038373 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce815d9a0 con 0x7fbce81542e0 2024-03-16T20:11:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.036+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce8070410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097c0 tx=0x7fbcd801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c91af340ec5441b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbce5ffb640 1 -- 172.21.15.32:0/3797038373 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80ad640 con 0x7fbce81542e0 2024-03-16T20:11:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbce5ffb640 1 -- 172.21.15.32:0/3797038373 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd80ad7a0 con 0x7fbce81542e0 2024-03-16T20:11:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbce5ffb640 1 -- 172.21.15.32:0/3797038373 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80adb30 con 0x7fbce81542e0 2024-03-16T20:11:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 -- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 msgr2=0x7fbce8070410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce8070410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097c0 tx=0x7fbcd801c4a0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 -- 172.21.15.32:0/3797038373 shutdown_connections 2024-03-16T20:11:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce815a1d0 0x7fbce815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 --2- 172.21.15.32:0/3797038373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce8070410 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 -- 172.21.15.32:0/3797038373 >> 172.21.15.32:0/3797038373 conn(0x7fbce80c92d0 msgr2=0x7fbce80c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 -- 172.21.15.32:0/3797038373 shutdown_connections 2024-03-16T20:11:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.044+0000 7fbced761640 1 -- 172.21.15.32:0/3797038373 wait complete. 2024-03-16T20:11:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 Processor -- start 2024-03-16T20:11:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 -- start start 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce8073090 0x7fbce81f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce814b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81503c0 con 0x7fbce815a1d0 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81f6f50 con 0x7fbce81542e0 2024-03-16T20:11:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbced761640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8150680 con 0x7fbce8073090 2024-03-16T20:11:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbce77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbce77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58080/0 (socket says 172.21.15.32:58080) 2024-03-16T20:11:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbce77fe640 1 -- 172.21.15.32:0/322779437 learned_addr learned my addr 172.21.15.32:0/322779437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce8073090 0x7fbce81f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.048+0000 7fbce67fc640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce814b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.052+0000 7fbce77fe640 1 -- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce8073090 msgr2=0x7fbce81f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.052+0000 7fbce77fe640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce8073090 0x7fbce81f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.052+0000 7fbce77fe640 1 -- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 msgr2=0x7fbce814b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.052+0000 7fbce77fe640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce814b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.052+0000 7fbce77fe640 1 -- 172.21.15.32:0/322779437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8009420 con 0x7fbce815a1d0 2024-03-16T20:11:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.056+0000 7fbce77fe640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbce000efc0 tx=0x7fbce000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.060+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0014480 con 0x7fbce815a1d0 2024-03-16T20:11:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.060+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce000f040 con 0x7fbce815a1d0 2024-03-16T20:11:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.060+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0014820 con 0x7fbce815a1d0 2024-03-16T20:11:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.060+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce8150960 con 0x7fbce815a1d0 2024-03-16T20:11:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.064+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce81f64f0 con 0x7fbce815a1d0 2024-03-16T20:11:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.064+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8150b80 con 0x7fbce815a1d0 2024-03-16T20:11:42.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.352+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbce0014980 con 0x7fbce815a1d0 2024-03-16T20:11:42.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.352+0000 7fbccf7fe640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 0x7fbcd4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.352+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 0x7fbcd4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.352+0000 7fbce6ffd640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 0x7fbcd4069380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80af000 tx=0x7fbcd80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.364+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fbce0056860 con 0x7fbce815a1d0 2024-03-16T20:11:42.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.364+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fbce0052b00 con 0x7fbce815a1d0 2024-03-16T20:11:42.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.532+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbce80c7610 con 0x7fbcd4066ec0 2024-03-16T20:11:42.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.532+0000 7fbccf7fe640 1 -- 172.21.15.32:0/322779437 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbcd80a6040 con 0x7fbcd4066ec0 2024-03-16T20:11:42.526 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 0 to deep-scrub 2024-03-16T20:11:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 msgr2=0x7fbcd4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 0x7fbcd4069380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80af000 tx=0x7fbcd80a7070 comp rx=0 tx=0).stop 2024-03-16T20:11:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 msgr2=0x7fbce8147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbce000efc0 tx=0x7fbce000c490 comp rx=0 tx=0).stop 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 shutdown_connections 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbcd4066ec0 0x7fbcd4069380 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbce815a1d0 0x7fbce8147120 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbce81542e0 0x7fbce814b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 --2- 172.21.15.32:0/322779437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbce8073090 0x7fbce81f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 >> 172.21.15.32:0/322779437 conn(0x7fbce80c92d0 msgr2=0x7fbce80c96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:42.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 shutdown_connections 2024-03-16T20:11:42.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.536+0000 7fbced761640 1 -- 172.21.15.32:0/322779437 wait complete. 2024-03-16T20:11:42.545 INFO:tasks.ceph:Scrubbing osd.1 2024-03-16T20:11:42.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:42.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.660+0000 7f995c6b3640 1 Processor -- start 2024-03-16T20:11:42.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.660+0000 7f995c6b3640 1 -- start start 2024-03-16T20:11:42.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99541521e0 0x7f99540cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f995406fac0 0x7f99540cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995406f6a0 con 0x7f99541521e0 2024-03-16T20:11:42.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99540c3360 con 0x7f995406fac0 2024-03-16T20:11:42.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995c6b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540ca3c0 con 0x7f9954151e00 2024-03-16T20:11:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995a44f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995ac50640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f995406fac0 0x7f99540cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f9959c4e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99541521e0 0x7f99540cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995a44f640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44774/0 (socket says 172.21.15.32:44774) 2024-03-16T20:11:42.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.664+0000 7f995a44f640 1 -- 172.21.15.32:0/3035237960 learned_addr learned my addr 172.21.15.32:0/3035237960 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.668+0000 7f995a44f640 1 -- 172.21.15.32:0/3035237960 >> v2:172.21.15.59:3300/0 conn(0x7f995406fac0 msgr2=0x7f99540cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.668+0000 7f995a44f640 1 --2- 172.21.15.32:0/3035237960 >> v2:172.21.15.59:3300/0 conn(0x7f995406fac0 0x7f99540cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.668+0000 7f995a44f640 1 -- 172.21.15.32:0/3035237960 >> v2:172.21.15.32:3300/0 conn(0x7f99541521e0 msgr2=0x7f99540cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.668+0000 7f995a44f640 1 --2- 172.21.15.32:0/3035237960 >> v2:172.21.15.32:3300/0 conn(0x7f99541521e0 0x7f99540cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.668+0000 7f995a44f640 1 -- 172.21.15.32:0/3035237960 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540c88d0 con 0x7f9954151e00 2024-03-16T20:11:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.676+0000 7f995a44f640 1 --2- 172.21.15.32:0/3035237960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f99540c4b40 tx=0x7f995001c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaa86b0f837a9d39 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.684+0000 7f995944d640 1 -- 172.21.15.32:0/3035237960 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500ab600 con 0x7f9954151e00 2024-03-16T20:11:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.684+0000 7f995944d640 1 -- 172.21.15.32:0/3035237960 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99500a7050 con 0x7f9954151e00 2024-03-16T20:11:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.684+0000 7f995944d640 1 -- 172.21.15.32:0/3035237960 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500ab980 con 0x7f9954151e00 2024-03-16T20:11:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- 172.21.15.32:0/3035237960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 msgr2=0x7f99540ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3035237960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f99540c4b40 tx=0x7f995001c520 comp rx=0 tx=0).stop 2024-03-16T20:11:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- 172.21.15.32:0/3035237960 shutdown_connections 2024-03-16T20:11:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3035237960 >> v2:172.21.15.59:3300/0 conn(0x7f995406fac0 0x7f99540cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3035237960 >> v2:172.21.15.32:3300/0 conn(0x7f99541521e0 0x7f99540cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3035237960 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9954151e00 0x7f99540ca810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- 172.21.15.32:0/3035237960 >> 172.21.15.32:0/3035237960 conn(0x7f995406b5c0 msgr2=0x7f9954066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- 172.21.15.32:0/3035237960 shutdown_connections 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- 172.21.15.32:0/3035237960 wait complete. 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 Processor -- start 2024-03-16T20:11:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- start start 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995406fac0 0x7f99540cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 0x7f99541f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99541521e0 0x7f99541f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9954154900 con 0x7f9954151e00 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99541f7c70 con 0x7f995406fac0 2024-03-16T20:11:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995c6b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99541f70f0 con 0x7f99541521e0 2024-03-16T20:11:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f9959c4e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 0x7f99541f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f9959c4e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 0x7f99541f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58096/0 (socket says 172.21.15.32:58096) 2024-03-16T20:11:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f9959c4e640 1 -- 172.21.15.32:0/3087178281 learned_addr learned my addr 172.21.15.32:0/3087178281 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.688+0000 7f995a44f640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995406fac0 0x7f99540cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f995ac50640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99541521e0 0x7f99541f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f9959c4e640 1 -- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99541521e0 msgr2=0x7f99541f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f9959c4e640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99541521e0 0x7f99541f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f9959c4e640 1 -- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995406fac0 msgr2=0x7f99540cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f9959c4e640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995406fac0 0x7f99540cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.692+0000 7f9959c4e640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500094d0 con 0x7f9954151e00 2024-03-16T20:11:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.696+0000 7f9959c4e640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 0x7f99541f3ee0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f994400e930 tx=0x7f994400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944009690 con 0x7f9954151e00 2024-03-16T20:11:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994400ca00 con 0x7f9954151e00 2024-03-16T20:11:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400cd70 con 0x7f9954151e00 2024-03-16T20:11:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99541f73d0 con 0x7f9954151e00 2024-03-16T20:11:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99541f7990 con 0x7f9954151e00 2024-03-16T20:11:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.700+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99540b3b80 con 0x7f9954151e00 2024-03-16T20:11:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.728+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9944013470 con 0x7f9954151e00 2024-03-16T20:11:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.728+0000 7f994affd640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9938066e90 0x7f9938069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.728+0000 7f995a44f640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9938066e90 0x7f9938069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.728+0000 7f995a44f640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9938066e90 0x7f9938069350 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f99500ad000 tx=0x7f9950008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9944055c80 con 0x7f9954151e00 2024-03-16T20:11:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f994affd640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f9938070840 0x7f9938072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99380733b0 con 0x7f9938070840 2024-03-16T20:11:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9944051f20 con 0x7f9954151e00 2024-03-16T20:11:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f995ac50640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f9938070840 0x7f9938072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.732+0000 7f995ac50640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f9938070840 0x7f9938072ce0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.736+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== osd.1 v2:172.21.15.32:6802/1994708459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f99380733b0 con 0x7f9938070840 2024-03-16T20:11:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.760+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 --> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f99540c3620 con 0x7f9938070840 2024-03-16T20:11:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.760+0000 7f994affd640 1 -- 172.21.15.32:0/3087178281 <== osd.1 v2:172.21.15.32:6802/1994708459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f99540c3620 con 0x7f9938070840 2024-03-16T20:11:42.755 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:42.755 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f9938070840 msgr2=0x7f9938072ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6802/1994708459,v1:172.21.15.32:6803/1994708459] conn(0x7f9938070840 0x7f9938072ce0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9938066e90 msgr2=0x7f9938069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9938066e90 0x7f9938069350 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f99500ad000 tx=0x7f9950008040 comp rx=0 tx=0).stop 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 msgr2=0x7f99541f3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 --2- 172.21.15.32:0/3087178281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9954151e00 0x7f99541f3ee0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f994400e930 tx=0x7f994400edf0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995ac50640 1 -- 172.21.15.32:0/3087178281 reap_dead start 2024-03-16T20:11:42.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 shutdown_connections 2024-03-16T20:11:42.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 >> 172.21.15.32:0/3087178281 conn(0x7f995406b5c0 msgr2=0x7f99540cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:42.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 shutdown_connections 2024-03-16T20:11:42.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.764+0000 7f995c6b3640 1 -- 172.21.15.32:0/3087178281 wait complete. 2024-03-16T20:11:42.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-16T20:11:42.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.884+0000 7f4d21f3d640 1 Processor -- start 2024-03-16T20:11:42.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.884+0000 7f4d21f3d640 1 -- start start 2024-03-16T20:11:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.884+0000 7f4d21f3d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c0c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.884+0000 7f4d21f3d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d1c151f80 0x7f4d1c157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d21f3d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c1581f0 con 0x7f4d1c0c8170 2024-03-16T20:11:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d21f3d640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d1c0c9250 con 0x7f4d1c151f80 2024-03-16T20:11:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d21f3d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0c28a0 con 0x7f4d1c151ba0 2024-03-16T20:11:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d1affd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d1c151f80 0x7f4d1c157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d1affd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d1c151f80 0x7f4d1c157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58108/0 (socket says 172.21.15.32:58108) 2024-03-16T20:11:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d1affd640 1 -- 172.21.15.32:0/1197409444 learned_addr learned my addr 172.21.15.32:0/1197409444 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.888+0000 7f4d1b7fe640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c0c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.892+0000 7f4d1affd640 1 -- 172.21.15.32:0/1197409444 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 msgr2=0x7f4d1c0c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.892+0000 7f4d1affd640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c0c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.892+0000 7f4d1affd640 1 -- 172.21.15.32:0/1197409444 >> v1:172.21.15.59:6789/0 conn(0x7f4d1c0c8170 legacy=0x7f4d1c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.892+0000 7f4d1affd640 1 -- 172.21.15.32:0/1197409444 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c1523c0 con 0x7f4d1c151f80 2024-03-16T20:11:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.896+0000 7f4d1affd640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c151f80 0x7f4d1c157cb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d1001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262b1c0517cd52e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d1a7fc640 1 -- 172.21.15.32:0/1197409444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100ab640 con 0x7f4d1c151f80 2024-03-16T20:11:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d1a7fc640 1 -- 172.21.15.32:0/1197409444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d100a7070 con 0x7f4d1c151f80 2024-03-16T20:11:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d1a7fc640 1 -- 172.21.15.32:0/1197409444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100ab9c0 con 0x7f4d1c151f80 2024-03-16T20:11:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 -- 172.21.15.32:0/1197409444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c151f80 msgr2=0x7f4d1c157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c151f80 0x7f4d1c157cb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d1001c530 comp rx=0 tx=0).stop 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 -- 172.21.15.32:0/1197409444 shutdown_connections 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c151f80 0x7f4d1c157cb0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/1197409444 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c0c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 -- 172.21.15.32:0/1197409444 >> 172.21.15.32:0/1197409444 conn(0x7f4d1c06b6b0 msgr2=0x7f4d1c066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 -- 172.21.15.32:0/1197409444 shutdown_connections 2024-03-16T20:11:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.900+0000 7f4d21f3d640 1 -- 172.21.15.32:0/1197409444 wait complete. 2024-03-16T20:11:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 Processor -- start 2024-03-16T20:11:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 -- start start 2024-03-16T20:11:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c14eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c151f80 0x7f4d1c0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1f0bd0 con 0x7f4d1c0c8170 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1f13f0 con 0x7f4d1c151f80 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d21f3d640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c14f0b0 con 0x7f4d1c151ba0 2024-03-16T20:11:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d1b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d1b7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58120/0 (socket says 172.21.15.32:58120) 2024-03-16T20:11:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d1b7fe640 1 -- 172.21.15.32:0/3052298086 learned_addr learned my addr 172.21.15.32:0/3052298086 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d1affd640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c14eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.904+0000 7f4d1bfff640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c151f80 0x7f4d1c0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.908+0000 7f4d1b7fe640 1 -- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 msgr2=0x7f4d1c14eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.908+0000 7f4d1b7fe640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c14eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.908+0000 7f4d1b7fe640 1 -- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c151f80 msgr2=0x7f4d1c0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:42.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.908+0000 7f4d1b7fe640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c151f80 0x7f4d1c0c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:42.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.912+0000 7f4d1b7fe640 1 -- 172.21.15.32:0/3052298086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100094d0 con 0x7f4d1c0c8170 2024-03-16T20:11:42.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.912+0000 7f4d1b7fe640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00e930 tx=0x7f4d0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c009690 con 0x7f4d1c0c8170 2024-03-16T20:11:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c00ca00 con 0x7f4d1c0c8170 2024-03-16T20:11:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c1f0dc0 con 0x7f4d1c0c8170 2024-03-16T20:11:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c00cd70 con 0x7f4d1c0c8170 2024-03-16T20:11:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c1f4e30 con 0x7f4d1c0c8170 2024-03-16T20:11:42.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.920+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c068000 con 0x7f4d1c0c8170 2024-03-16T20:11:42.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.944+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4d0c013470 con 0x7f4d1c0c8170 2024-03-16T20:11:42.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.944+0000 7f4d03fff640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 0x7f4d08069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:42.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.944+0000 7f4d1affd640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 0x7f4d08069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:42.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.944+0000 7f4d1affd640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 0x7f4d08069350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4d100af000 tx=0x7f4d100a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:42.952+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f4d0c056110 con 0x7f4d1c0c8170 2024-03-16T20:11:43.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:11:43.138 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:11:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.224+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f4d0c0523b0 con 0x7f4d1c0c8170 2024-03-16T20:11:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.432+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d1c0c3910 con 0x7f4d08066e90 2024-03-16T20:11:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.432+0000 7f4d03fff640 1 -- 172.21.15.32:0/3052298086 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4d1c0c3910 con 0x7f4d08066e90 2024-03-16T20:11:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 1 to deep-scrub 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 msgr2=0x7f4d08069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 0x7f4d08069350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4d100af000 tx=0x7f4d100a7000 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 msgr2=0x7f4d1c0ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00e930 tx=0x7f4d0c00edf0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 shutdown_connections 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4d08066e90 0x7f4d08069350 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c151f80 0x7f4d1c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d1c151ba0 0x7f4d1c14eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 --2- 172.21.15.32:0/3052298086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d1c0c8170 0x7f4d1c0ccba0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 >> 172.21.15.32:0/3052298086 conn(0x7f4d1c06b6b0 msgr2=0x7f4d1c155c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 shutdown_connections 2024-03-16T20:11:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.436+0000 7f4d21f3d640 1 -- 172.21.15.32:0/3052298086 wait complete. 2024-03-16T20:11:43.451 INFO:tasks.ceph:Scrubbing osd.2 2024-03-16T20:11:43.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:43.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 Processor -- start 2024-03-16T20:11:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 -- start start 2024-03-16T20:11:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60c4150de0 0x7f60c40c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c415a140 0x7f60c40c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c40c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c415a560 con 0x7f60c415a140 2024-03-16T20:11:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c4072850 con 0x7f60c4150de0 2024-03-16T20:11:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c8ab7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40c7910 con 0x7f60c40c8740 2024-03-16T20:11:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.676+0000 7f60c1d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c415a140 0x7f60c40c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c1d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c415a140 0x7f60c40c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58122/0 (socket says 172.21.15.32:58122) 2024-03-16T20:11:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c1d9b640 1 -- 172.21.15.32:0/1845025043 learned_addr learned my addr 172.21.15.32:0/1845025043 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c259c640 1 --2- 172.21.15.32:0/1845025043 >> v2:172.21.15.59:3300/0 conn(0x7f60c4150de0 0x7f60c40c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c2d9d640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c40c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c1d9b640 1 -- 172.21.15.32:0/1845025043 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 msgr2=0x7f60c40c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c1d9b640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c40c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.680+0000 7f60c1d9b640 1 -- 172.21.15.32:0/1845025043 >> v2:172.21.15.59:3300/0 conn(0x7f60c4150de0 msgr2=0x7f60c40c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.684+0000 7f60c1d9b640 1 --2- 172.21.15.32:0/1845025043 >> v2:172.21.15.59:3300/0 conn(0x7f60c4150de0 0x7f60c40c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.684+0000 7f60c1d9b640 1 -- 172.21.15.32:0/1845025043 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40c7bf0 con 0x7f60c415a140 2024-03-16T20:11:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.688+0000 7f60c1d9b640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c40c8080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f60c406cf10 tx=0x7f60b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392b29029c7721cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.692+0000 7f60c159a640 1 -- 172.21.15.32:0/1845025043 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80ab600 con 0x7f60c415a140 2024-03-16T20:11:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.692+0000 7f60c159a640 1 -- 172.21.15.32:0/1845025043 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b80a7050 con 0x7f60c415a140 2024-03-16T20:11:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.692+0000 7f60c159a640 1 -- 172.21.15.32:0/1845025043 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80ab980 con 0x7f60c415a140 2024-03-16T20:11:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- 172.21.15.32:0/1845025043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 msgr2=0x7f60c40c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c40c8080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f60c406cf10 tx=0x7f60b801c490 comp rx=0 tx=0).stop 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- 172.21.15.32:0/1845025043 shutdown_connections 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c40c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/1845025043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c40c8080 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/1845025043 >> v2:172.21.15.59:3300/0 conn(0x7f60c4150de0 0x7f60c40c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- 172.21.15.32:0/1845025043 >> 172.21.15.32:0/1845025043 conn(0x7f60c40cc7a0 msgr2=0x7f60c40ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- 172.21.15.32:0/1845025043 shutdown_connections 2024-03-16T20:11:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- 172.21.15.32:0/1845025043 wait complete. 2024-03-16T20:11:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 Processor -- start 2024-03-16T20:11:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- start start 2024-03-16T20:11:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c4142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60c4150de0 0x7f60c41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c413abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c413efd0 con 0x7f60c415a140 2024-03-16T20:11:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4158f50 con 0x7f60c4150de0 2024-03-16T20:11:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c8ab7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4159510 con 0x7f60c40c8740 2024-03-16T20:11:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c413abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c259c640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c4142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60c4150de0 0x7f60c41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c2d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c413abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58132/0 (socket says 172.21.15.32:58132) 2024-03-16T20:11:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.696+0000 7f60c2d9d640 1 -- 172.21.15.32:0/547477583 learned_addr learned my addr 172.21.15.32:0/547477583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.700+0000 7f60c2d9d640 1 -- 172.21.15.32:0/547477583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 msgr2=0x7f60c4142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.700+0000 7f60c2d9d640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f60c40c8740 0x7f60c4142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.700+0000 7f60c2d9d640 1 -- 172.21.15.32:0/547477583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60c4150de0 msgr2=0x7f60c41599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.700+0000 7f60c2d9d640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60c4150de0 0x7f60c41599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.700+0000 7f60c2d9d640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b80094d0 con 0x7f60c415a140 2024-03-16T20:11:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.704+0000 7f60c2d9d640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c413abf0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00efc0 tx=0x7f60bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.708+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc009ba0 con 0x7f60c415a140 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.708+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc00f040 con 0x7f60c415a140 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.708+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c413bf30 con 0x7f60c415a140 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.708+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc0146c0 con 0x7f60c415a140 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.712+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c413c1b0 con 0x7f60c415a140 2024-03-16T20:11:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.712+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60c40c3770 con 0x7f60c415a140 2024-03-16T20:11:43.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.732+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60bc014820 con 0x7f60c415a140 2024-03-16T20:11:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.732+0000 7f60b2ffd640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60a4066e90 0x7f60a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.732+0000 7f60c259c640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60a4066e90 0x7f60a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60c259c640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60a4066e90 0x7f60a4069350 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009a40 tx=0x7f60b4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f60bc056700 con 0x7f60c415a140 2024-03-16T20:11:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60b2ffd640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f60a40709b0 0x7f60a4072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60a4073520 con 0x7f60a40709b0 2024-03-16T20:11:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60bc0529a0 con 0x7f60c415a140 2024-03-16T20:11:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60c1d9b640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f60a40709b0 0x7f60a4072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60c1d9b640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f60a40709b0 0x7f60a4072e50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.736+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== osd.2 v2:172.21.15.32:6818/3435819214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f60a4073520 con 0x7f60a40709b0 2024-03-16T20:11:43.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 --> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f60c4068000 con 0x7f60a40709b0 2024-03-16T20:11:43.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60b2ffd640 1 -- 172.21.15.32:0/547477583 <== osd.2 v2:172.21.15.32:6818/3435819214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f60c4068000 con 0x7f60a40709b0 2024-03-16T20:11:43.745 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:43.745 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:43.745 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:43.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f60a40709b0 msgr2=0x7f60a4072e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6818/3435819214,v1:172.21.15.32:6819/3435819214] conn(0x7f60a40709b0 0x7f60a4072e50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60a4066e90 msgr2=0x7f60a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f60a4066e90 0x7f60a4069350 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009a40 tx=0x7f60b4009190 comp rx=0 tx=0).stop 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 msgr2=0x7f60c413abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 --2- 172.21.15.32:0/547477583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c415a140 0x7f60c413abf0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00efc0 tx=0x7f60bc00c490 comp rx=0 tx=0).stop 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c2d9d640 1 -- 172.21.15.32:0/547477583 reap_dead start 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 shutdown_connections 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 >> 172.21.15.32:0/547477583 conn(0x7f60c40cc7a0 msgr2=0x7f60c40c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 shutdown_connections 2024-03-16T20:11:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.752+0000 7f60c8ab7640 1 -- 172.21.15.32:0/547477583 wait complete. 2024-03-16T20:11:43.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 Processor -- start 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 -- start start 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 0x7f9e580c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5815bdf0 con 0x7f9e5806f0a0 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e58152230 con 0x7f9e580c97b0 2024-03-16T20:11:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e5e5d9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58158d30 con 0x7f9e58151e50 2024-03-16T20:11:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e577fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e5806f0a0 0x7f9e580c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51276/0 (socket says 172.21.15.32:51276) 2024-03-16T20:11:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.876+0000 7f9e577fe640 1 -- 172.21.15.32:0/3286607675 learned_addr learned my addr 172.21.15.32:0/3286607675 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.880+0000 7f9e57fff640 1 --2- 172.21.15.32:0/3286607675 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 0x7f9e580c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.880+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008872227 0 0) 0x7f9e5815bdf0 con 0x7f9e5806f0a0 2024-03-16T20:11:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.880+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e40003530 con 0x7f9e5806f0a0 2024-03-16T20:11:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.884+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 <== mon.1 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49784640 0 0) 0x7f9e58152230 con 0x7f9e580c97b0 2024-03-16T20:11:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.884+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5815bdf0 con 0x7f9e580c97b0 2024-03-16T20:11:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.888+0000 7f9e57fff640 1 -- 172.21.15.32:0/3286607675 >> v1:172.21.15.59:6789/0 conn(0x7f9e580c97b0 legacy=0x7f9e58157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.888+0000 7f9e57fff640 1 -- 172.21.15.32:0/3286607675 >> v1:172.21.15.32:6789/0 conn(0x7f9e5806f0a0 legacy=0x7f9e580c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.888+0000 7f9e57fff640 1 -- 172.21.15.32:0/3286607675 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5815c280 con 0x7f9e58151e50 2024-03-16T20:11:43.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.892+0000 7f9e57fff640 1 --2- 172.21.15.32:0/3286607675 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 0x7f9e580c9270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e480097f0 tx=0x7f9e4801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf1c95643ef4a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480a89d0 con 0x7f9e58151e50 2024-03-16T20:11:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e480a8b30 con 0x7f9e58151e50 2024-03-16T20:11:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e56ffd640 1 -- 172.21.15.32:0/3286607675 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480b0440 con 0x7f9e58151e50 2024-03-16T20:11:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3286607675 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 msgr2=0x7f9e580c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3286607675 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 0x7f9e580c9270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e480097f0 tx=0x7f9e4801ecd0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3286607675 shutdown_connections 2024-03-16T20:11:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3286607675 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e58151e50 0x7f9e580c9270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3286607675 >> 172.21.15.32:0/3286607675 conn(0x7f9e58070f90 msgr2=0x7f9e580713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3286607675 shutdown_connections 2024-03-16T20:11:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.900+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3286607675 wait complete. 2024-03-16T20:11:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 Processor -- start 2024-03-16T20:11:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 -- start start 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e5806f0a0 0x7f9e580c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e58151e50 0x7f9e580c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580c6910 con 0x7f9e580c97b0 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5814b610 con 0x7f9e58151e50 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5e5d9640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580b14b0 con 0x7f9e5806f0a0 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e577fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e577fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58144/0 (socket says 172.21.15.32:58144) 2024-03-16T20:11:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e577fe640 1 -- 172.21.15.32:0/3475851630 learned_addr learned my addr 172.21.15.32:0/3475851630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.904+0000 7f9e5cb76640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e58151e50 0x7f9e580c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.908+0000 7f9e57fff640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e5806f0a0 0x7f9e580c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 -- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e5806f0a0 msgr2=0x7f9e580c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e5806f0a0 0x7f9e580c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 -- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e58151e50 msgr2=0x7f9e580c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e58151e50 0x7f9e580c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 -- 172.21.15.32:0/3475851630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48009420 con 0x7f9e580c97b0 2024-03-16T20:11:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.912+0000 7f9e577fe640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c4d0 tx=0x7f9e4c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.916+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c005cf0 con 0x7f9e580c97b0 2024-03-16T20:11:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.916+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c005e50 con 0x7f9e580c97b0 2024-03-16T20:11:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.920+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e580b1750 con 0x7f9e580c97b0 2024-03-16T20:11:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.920+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0072b0 con 0x7f9e580c97b0 2024-03-16T20:11:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.920+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e581ef900 con 0x7f9e580c97b0 2024-03-16T20:11:43.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:43.920+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e58068000 con 0x7f9e580c97b0 2024-03-16T20:11:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.184+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9e4c007410 con 0x7f9e580c97b0 2024-03-16T20:11:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.184+0000 7f9e54ff9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 0x7f9e44069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.184+0000 7f9e57fff640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 0x7f9e44069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.184+0000 7f9e57fff640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 0x7f9e44069300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e480b2000 tx=0x7f9e48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.196+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f9e4c055050 con 0x7f9e580c97b0 2024-03-16T20:11:44.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.196+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f9e4c0512f0 con 0x7f9e580c97b0 2024-03-16T20:11:44.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.388+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e5815b4c0 con 0x7f9e44066e40 2024-03-16T20:11:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.388+0000 7f9e54ff9640 1 -- 172.21.15.32:0/3475851630 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9e480a8970 con 0x7f9e44066e40 2024-03-16T20:11:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 2 to deep-scrub 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 msgr2=0x7f9e44069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 0x7f9e44069300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e480b2000 tx=0x7f9e48008040 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 msgr2=0x7f9e581f0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c4d0 tx=0x7f9e4c00c990 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 shutdown_connections 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f9e44066e40 0x7f9e44069300 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e58151e50 0x7f9e580c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580c97b0 0x7f9e581f0100 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 --2- 172.21.15.32:0/3475851630 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e5806f0a0 0x7f9e580c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 >> 172.21.15.32:0/3475851630 conn(0x7f9e58070f90 msgr2=0x7f9e5815aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:44.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.392+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 shutdown_connections 2024-03-16T20:11:44.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.396+0000 7f9e5e5d9640 1 -- 172.21.15.32:0/3475851630 wait complete. 2024-03-16T20:11:44.408 INFO:tasks.ceph:Scrubbing osd.3 2024-03-16T20:11:44.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 Processor -- start 2024-03-16T20:11:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 -- start start 2024-03-16T20:11:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f8110165940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811015d5a0 con 0x7f81100c8bb0 2024-03-16T20:11:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811015d880 con 0x7f81101543f0 2024-03-16T20:11:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.524+0000 7f8117a15640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811006ba40 con 0x7f81100c72f0 2024-03-16T20:11:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.528+0000 7f8115fb2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f8110165940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.528+0000 7f8115fb2640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f8110165940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44850/0 (socket says 172.21.15.32:44850) 2024-03-16T20:11:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.528+0000 7f8115fb2640 1 -- 172.21.15.32:0/470363857 learned_addr learned my addr 172.21.15.32:0/470363857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.596+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188696064 0 0) 0x7f811015d5a0 con 0x7f81100c8bb0 2024-03-16T20:11:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.596+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc003530 con 0x7f81100c8bb0 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438218981 0 0) 0x7f80fc003530 con 0x7f81100c8bb0 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 msgr2=0x7f8110165940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f8110165940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 >> v1:172.21.15.59:6789/0 conn(0x7f81101543f0 legacy=0x7f811015dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811015ae70 con 0x7f81100c8bb0 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f8100001680 con 0x7f81100c8bb0 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 >> v1:172.21.15.32:6789/0 conn(0x7f81100c8bb0 legacy=0x7f811006f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80fc0039b0 0x7f80fc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80fc0047d0 0x7f80fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f80fc005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811006ba40 con 0x7f80fc0039b0 2024-03-16T20:11:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bbb0 con 0x7f80fc0047d0 2024-03-16T20:11:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bd40 con 0x7f81100c72f0 2024-03-16T20:11:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.600+0000 7f81157b1640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80fc0039b0 0x7f80fc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.604+0000 7f81157b1640 1 -- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 msgr2=0x7f80fc005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.604+0000 7f81157b1640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100c72f0 0x7f80fc005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.604+0000 7f81157b1640 1 -- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80fc0047d0 msgr2=0x7f80fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.604+0000 7f81157b1640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80fc0047d0 0x7f80fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.604+0000 7f81157b1640 1 -- 172.21.15.32:0/470363857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc003530 con 0x7f80fc0039b0 2024-03-16T20:11:44.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.612+0000 7f81157b1640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80fc0039b0 0x7f80fc003db0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f810c001650 tx=0x7f810c01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10703be42f174e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c005270 con 0x7f80fc0039b0 2024-03-16T20:11:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c0a7070 con 0x7f80fc0039b0 2024-03-16T20:11:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8107fff640 1 -- 172.21.15.32:0/470363857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c003620 con 0x7f80fc0039b0 2024-03-16T20:11:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 -- 172.21.15.32:0/470363857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80fc0039b0 msgr2=0x7f80fc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 --2- 172.21.15.32:0/470363857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80fc0039b0 0x7f80fc003db0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f810c001650 tx=0x7f810c01c300 comp rx=0 tx=0).stop 2024-03-16T20:11:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8115fb2640 1 -- 172.21.15.32:0/470363857 reap_dead start 2024-03-16T20:11:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 -- 172.21.15.32:0/470363857 shutdown_connections 2024-03-16T20:11:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 -- 172.21.15.32:0/470363857 >> 172.21.15.32:0/470363857 conn(0x7f81100cc7a0 msgr2=0x7f81100ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 -- 172.21.15.32:0/470363857 shutdown_connections 2024-03-16T20:11:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.616+0000 7f8117a15640 1 -- 172.21.15.32:0/470363857 wait complete. 2024-03-16T20:11:44.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 Processor -- start 2024-03-16T20:11:44.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 -- start start 2024-03-16T20:11:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100cb680 0x7f811015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81100cba60 0x7f811015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 0x7f81101f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811015c2c0 con 0x7f81101f10c0 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811015b8f0 con 0x7f81100cba60 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8117a15640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8110156590 con 0x7f81100cb680 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.620+0000 7f8115fb2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 0x7f81101f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.624+0000 7f8114fb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81100cba60 0x7f811015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.624+0000 7f8115fb2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 0x7f81101f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58158/0 (socket says 172.21.15.32:58158) 2024-03-16T20:11:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.624+0000 7f8115fb2640 1 -- 172.21.15.32:0/4031938081 learned_addr learned my addr 172.21.15.32:0/4031938081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.628+0000 7f8115fb2640 1 -- 172.21.15.32:0/4031938081 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100cb680 msgr2=0x7f811015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.628+0000 7f8115fb2640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81100cb680 0x7f811015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.628+0000 7f8115fb2640 1 -- 172.21.15.32:0/4031938081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81100cba60 msgr2=0x7f811015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.628+0000 7f8115fb2640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81100cba60 0x7f811015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.628+0000 7f8115fb2640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110156150 con 0x7f81101f10c0 2024-03-16T20:11:44.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.636+0000 7f8115fb2640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 0x7f81101f31c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f811006f0a0 tx=0x7f8108007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108017070 con 0x7f81101f10c0 2024-03-16T20:11:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8110154c80 con 0x7f81101f10c0 2024-03-16T20:11:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811015b2a0 con 0x7f81101f10c0 2024-03-16T20:11:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810800f040 con 0x7f81101f10c0 2024-03-16T20:11:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810800da30 con 0x7f81101f10c0 2024-03-16T20:11:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.640+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80f0001040 con 0x7f81101f10c0 2024-03-16T20:11:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.672+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f810800db90 con 0x7f81101f10c0 2024-03-16T20:11:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.672+0000 7f8105ffb640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80f4066e40 0x7f80f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.672+0000 7f81157b1640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80f4066e40 0x7f80f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.672+0000 7f81157b1640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80f4066e40 0x7f80f4069300 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f810c0aa000 tx=0x7f810c000fc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f8108055e80 con 0x7f81101f10c0 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8105ffb640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f80f4070860 0x7f80f4072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80f40733d0 con 0x7f80f4070860 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8108051a20 con 0x7f81101f10c0 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8114fb0640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f80f4070860 0x7f80f4072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.680+0000 7f8114fb0640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f80f4070860 0x7f80f4072d00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.684+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== osd.3 v2:172.21.15.32:6826/1974622921 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f80f40733d0 con 0x7f80f4070860 2024-03-16T20:11:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 --> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f80f0002cc0 con 0x7f80f4070860 2024-03-16T20:11:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8105ffb640 1 -- 172.21.15.32:0/4031938081 <== osd.3 v2:172.21.15.32:6826/1974622921 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f80f0002cc0 con 0x7f80f4070860 2024-03-16T20:11:44.693 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:44.693 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:44.693 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:44.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f80f4070860 msgr2=0x7f80f4072d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6826/1974622921,v1:172.21.15.32:6827/1974622921] conn(0x7f80f4070860 0x7f80f4072d00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80f4066e40 msgr2=0x7f80f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f80f4066e40 0x7f80f4069300 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f810c0aa000 tx=0x7f810c000fc0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 msgr2=0x7f81101f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 --2- 172.21.15.32:0/4031938081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81101f10c0 0x7f81101f31c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f811006f0a0 tx=0x7f8108007cb0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8115fb2640 1 -- 172.21.15.32:0/4031938081 reap_dead start 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 shutdown_connections 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 >> 172.21.15.32:0/4031938081 conn(0x7f81100cc7a0 msgr2=0x7f81100ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 shutdown_connections 2024-03-16T20:11:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.700+0000 7f8117a15640 1 -- 172.21.15.32:0/4031938081 wait complete. 2024-03-16T20:11:44.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-16T20:11:44.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 Processor -- start 2024-03-16T20:11:44.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 -- start start 2024-03-16T20:11:44.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30c00c3540 0x7f30c00c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00c7e50 con 0x7f30c0150f00 2024-03-16T20:11:44.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c01512e0 con 0x7f30c00c7040 2024-03-16T20:11:44.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.820+0000 7f30c5fbc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c0070850 con 0x7f30c00c3540 2024-03-16T20:11:44.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.824+0000 7f30bf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30c0150f00 0x7f30c00c3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51300/0 (socket says 172.21.15.32:51300) 2024-03-16T20:11:44.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.824+0000 7f30bf7fe640 1 -- 172.21.15.32:0/3214345273 learned_addr learned my addr 172.21.15.32:0/3214345273 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:44.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.824+0000 7f30bffff640 1 --2- 172.21.15.32:0/3214345273 >> v2:172.21.15.59:3300/0 conn(0x7f30c00c3540 0x7f30c00c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.828+0000 7f30be7fc640 1 -- 172.21.15.32:0/3214345273 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586563128 0 0) 0x7f30c00c7e50 con 0x7f30c0150f00 2024-03-16T20:11:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.828+0000 7f30be7fc640 1 -- 172.21.15.32:0/3214345273 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a8003530 con 0x7f30c0150f00 2024-03-16T20:11:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.828+0000 7f30bffff640 1 -- 172.21.15.32:0/3214345273 >> v1:172.21.15.59:6789/0 conn(0x7f30c00c7040 legacy=0x7f30c015f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.828+0000 7f30bffff640 1 -- 172.21.15.32:0/3214345273 >> v1:172.21.15.32:6789/0 conn(0x7f30c0150f00 legacy=0x7f30c00c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.828+0000 7f30bffff640 1 -- 172.21.15.32:0/3214345273 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0070290 con 0x7f30c00c3540 2024-03-16T20:11:44.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.832+0000 7f30bffff640 1 --2- 172.21.15.32:0/3214345273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c00c3540 0x7f30c00c88d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009d90 tx=0x7f30b801f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=134bd32ce0fb6774 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30be7fc640 1 -- 172.21.15.32:0/3214345273 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80af600 con 0x7f30c00c3540 2024-03-16T20:11:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30be7fc640 1 -- 172.21.15.32:0/3214345273 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b80ab050 con 0x7f30c00c3540 2024-03-16T20:11:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30be7fc640 1 -- 172.21.15.32:0/3214345273 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80af980 con 0x7f30c00c3540 2024-03-16T20:11:44.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 -- 172.21.15.32:0/3214345273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c00c3540 msgr2=0x7f30c00c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/3214345273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c00c3540 0x7f30c00c88d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009d90 tx=0x7f30b801f120 comp rx=0 tx=0).stop 2024-03-16T20:11:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 -- 172.21.15.32:0/3214345273 shutdown_connections 2024-03-16T20:11:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/3214345273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c00c3540 0x7f30c00c88d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 -- 172.21.15.32:0/3214345273 >> 172.21.15.32:0/3214345273 conn(0x7f30c0066ea0 msgr2=0x7f30c00721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 -- 172.21.15.32:0/3214345273 shutdown_connections 2024-03-16T20:11:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.840+0000 7f30c5fbc640 1 -- 172.21.15.32:0/3214345273 wait complete. 2024-03-16T20:11:44.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 Processor -- start 2024-03-16T20:11:44.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 -- start start 2024-03-16T20:11:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30c00c3540 0x7f30c01468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0150f00 0x7f30c0159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0158850 con 0x7f30c00c7040 2024-03-16T20:11:44.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01433b0 con 0x7f30c0150f00 2024-03-16T20:11:44.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30c5fbc640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0142550 con 0x7f30c00c3540 2024-03-16T20:11:44.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30beffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30bf7fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30c00c3540 0x7f30c01468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30beffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58168/0 (socket says 172.21.15.32:58168) 2024-03-16T20:11:44.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30beffd640 1 -- 172.21.15.32:0/729260746 learned_addr learned my addr 172.21.15.32:0/729260746 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:44.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.844+0000 7f30bffff640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0150f00 0x7f30c0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:44.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.848+0000 7f30beffd640 1 -- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30c00c3540 msgr2=0x7f30c01468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.848+0000 7f30beffd640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30c00c3540 0x7f30c01468a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.848+0000 7f30beffd640 1 -- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0150f00 msgr2=0x7f30c0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:44.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.848+0000 7f30beffd640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0150f00 0x7f30c0159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:44.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.848+0000 7f30beffd640 1 -- 172.21.15.32:0/729260746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8009a70 con 0x7f30c00c7040 2024-03-16T20:11:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.872+0000 7f30beffd640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30b400c490 tx=0x7f30b4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.880+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4003600 con 0x7f30c00c7040 2024-03-16T20:11:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.880+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b4003780 con 0x7f30c00c7040 2024-03-16T20:11:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.880+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c0142830 con 0x7f30c00c7040 2024-03-16T20:11:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.880+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4003b30 con 0x7f30c00c7040 2024-03-16T20:11:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.880+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c0142df0 con 0x7f30c00c7040 2024-03-16T20:11:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:44.884+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c0068000 con 0x7f30c00c7040 2024-03-16T20:11:45.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.152+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f30b4003c90 con 0x7f30c00c7040 2024-03-16T20:11:45.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.156+0000 7f30a7fff640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 0x7f30ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.156+0000 7f30bf7fe640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 0x7f30ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.156+0000 7f30bf7fe640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 0x7f30ac069300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f30b0007fd0 tx=0x7f30b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.176+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f30b4055040 con 0x7f30c00c7040 2024-03-16T20:11:45.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.176+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f30b40512e0 con 0x7f30c00c7040 2024-03-16T20:11:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f30c00c7ae0 con 0x7f30ac066e40 2024-03-16T20:11:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30a7fff640 1 -- 172.21.15.32:0/729260746 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f30b0001480 con 0x7f30ac066e40 2024-03-16T20:11:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 3 to deep-scrub 2024-03-16T20:11:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 msgr2=0x7f30ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 0x7f30ac069300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f30b0007fd0 tx=0x7f30b0009040 comp rx=0 tx=0).stop 2024-03-16T20:11:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 msgr2=0x7f30c0158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30b400c490 tx=0x7f30b4001e80 comp rx=0 tx=0).stop 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 shutdown_connections 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f30ac066e40 0x7f30ac069300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.364+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30c0150f00 0x7f30c0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.368+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c00c7040 0x7f30c0158fe0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.368+0000 7f30c5fbc640 1 --2- 172.21.15.32:0/729260746 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f30c00c3540 0x7f30c01468a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.368+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 >> 172.21.15.32:0/729260746 conn(0x7f30c0066ea0 msgr2=0x7f30c014e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.368+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 shutdown_connections 2024-03-16T20:11:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.368+0000 7f30c5fbc640 1 -- 172.21.15.32:0/729260746 wait complete. 2024-03-16T20:11:45.374 INFO:tasks.ceph:Scrubbing osd.4 2024-03-16T20:11:45.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:45.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.496+0000 7f2a40606640 1 Processor -- start 2024-03-16T20:11:45.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.496+0000 7f2a40606640 1 -- start start 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a40606640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a3815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a40606640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3815d5a0 con 0x7f2a380c8bb0 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a40606640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3815d880 con 0x7f2a3815e250 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a40606640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3806ba40 con 0x7f2a381543f0 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a3dba1640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2a380c8bb0 0x7f2a3806f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51302/0 (socket says 172.21.15.32:51302) 2024-03-16T20:11:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.500+0000 7f2a3dba1640 1 -- 172.21.15.32:0/178920402 learned_addr learned my addr 172.21.15.32:0/178920402 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.504+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146823736 0 0) 0x7f2a3815d5a0 con 0x7f2a380c8bb0 2024-03-16T20:11:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.504+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a24003530 con 0x7f2a380c8bb0 2024-03-16T20:11:45.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3e3a2640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a3815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663094299 0 0) 0x7f2a24003530 con 0x7f2a380c8bb0 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 msgr2=0x7f2a3815dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a3815dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 >> v1:172.21.15.59:6789/0 conn(0x7f2a3815e250 legacy=0x7f2a38164930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3815ae70 con 0x7f2a380c8bb0 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7f2a28001680 con 0x7f2a380c8bb0 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 >> v1:172.21.15.32:6789/0 conn(0x7f2a380c8bb0 legacy=0x7f2a3806f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004030 0x7f2a240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a24004d10 0x7f2a24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3806ba40 con 0x7f2a24004030 2024-03-16T20:11:45.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2400bc60 con 0x7f2a24004d10 2024-03-16T20:11:45.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.508+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2400be20 con 0x7f2a381543f0 2024-03-16T20:11:45.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.512+0000 7f2a3dba1640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004030 0x7f2a240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.516+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a24004d10 0x7f2a24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.516+0000 7f2a3e3a2640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.524+0000 7f2a3eba3640 1 -- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 msgr2=0x7f2a240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:45.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.524+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a381543f0 0x7f2a240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.524+0000 7f2a3eba3640 1 -- 172.21.15.32:0/178920402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004030 msgr2=0x7f2a240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:45.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.524+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004030 0x7f2a240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.524+0000 7f2a3eba3640 1 -- 172.21.15.32:0/178920402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24003530 con 0x7f2a24004d10 2024-03-16T20:11:45.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.560+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a24004d10 0x7f2a24005190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a3815e660 tx=0x7f2a3001c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f43ef6fad299a23a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30003b80 con 0x7f2a24004d10 2024-03-16T20:11:45.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30003ce0 con 0x7f2a24004d10 2024-03-16T20:11:45.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a3d3a0640 1 -- 172.21.15.32:0/178920402 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a300052c0 con 0x7f2a24004d10 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 -- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a24004d10 msgr2=0x7f2a24005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 --2- 172.21.15.32:0/178920402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a24004d10 0x7f2a24005190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a3815e660 tx=0x7f2a3001c720 comp rx=0 tx=0).stop 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a3eba3640 1 -- 172.21.15.32:0/178920402 reap_dead start 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 -- 172.21.15.32:0/178920402 shutdown_connections 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 -- 172.21.15.32:0/178920402 >> 172.21.15.32:0/178920402 conn(0x7f2a380cc7a0 msgr2=0x7f2a380ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 -- 172.21.15.32:0/178920402 shutdown_connections 2024-03-16T20:11:45.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.672+0000 7f2a40606640 1 -- 172.21.15.32:0/178920402 wait complete. 2024-03-16T20:11:45.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 Processor -- start 2024-03-16T20:11:45.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 -- start start 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004d10 0x7f2a380c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a380c4ad0 0x7f2a380cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 0x7f2a380cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381548a0 con 0x7f2a24004d10 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380c60b0 con 0x7f2a380c4ad0 2024-03-16T20:11:45.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a40606640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38154ab0 con 0x7f2a38159810 2024-03-16T20:11:45.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a3eba3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 0x7f2a380cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a3eba3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 0x7f2a380cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44892/0 (socket says 172.21.15.32:44892) 2024-03-16T20:11:45.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.676+0000 7f2a3eba3640 1 -- 172.21.15.32:0/3211152968 learned_addr learned my addr 172.21.15.32:0/3211152968 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:45.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.684+0000 7f2a3eba3640 1 -- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a380c4ad0 msgr2=0x7f2a380cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.684+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a380c4ad0 0x7f2a380cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.684+0000 7f2a3eba3640 1 -- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004d10 msgr2=0x7f2a380c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.684+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a24004d10 0x7f2a380c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.684+0000 7f2a3eba3640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a38154d90 con 0x7f2a38159810 2024-03-16T20:11:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.688+0000 7f2a3eba3640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 0x7f2a380cbc80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a30098130 tx=0x7f2a24005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30006d60 con 0x7f2a38159810 2024-03-16T20:11:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30007dc0 con 0x7f2a38159810 2024-03-16T20:11:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a40606640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a380c6300 con 0x7f2a38159810 2024-03-16T20:11:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30007250 con 0x7f2a38159810 2024-03-16T20:11:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a40606640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a380c5cb0 con 0x7f2a38159810 2024-03-16T20:11:45.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.696+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a14001040 con 0x7f2a38159810 2024-03-16T20:11:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.736+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2a300073b0 con 0x7f2a38159810 2024-03-16T20:11:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.736+0000 7f2a2effd640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a1c066e40 0x7f2a1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.736+0000 7f2a3e3a2640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a1c066e40 0x7f2a1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.736+0000 7f2a3e3a2640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a1c066e40 0x7f2a1c069300 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006aa0 tx=0x7f2a34006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.740+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f2a30005820 con 0x7f2a38159810 2024-03-16T20:11:45.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.740+0000 7f2a2effd640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f2a1c070870 0x7f2a1c072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.740+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a1c0733e0 con 0x7f2a1c070870 2024-03-16T20:11:45.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.740+0000 7f2a3dba1640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f2a1c070870 0x7f2a1c072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.744+0000 7f2a3dba1640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f2a1c070870 0x7f2a1c072d10 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.744+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== osd.4 v2:172.21.15.59:6824/446247080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2a1c0733e0 con 0x7f2a1c070870 2024-03-16T20:11:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 --> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2a14002da0 con 0x7f2a1c070870 2024-03-16T20:11:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2effd640 1 -- 172.21.15.32:0/3211152968 <== osd.4 v2:172.21.15.59:6824/446247080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2a14002da0 con 0x7f2a1c070870 2024-03-16T20:11:45.751 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:45.751 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:45.751 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f2a1c070870 msgr2=0x7f2a1c072d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:6824/446247080,v1:172.21.15.59:6825/446247080] conn(0x7f2a1c070870 0x7f2a1c072d10 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a1c066e40 msgr2=0x7f2a1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f2a1c066e40 0x7f2a1c069300 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006aa0 tx=0x7f2a34006a30 comp rx=0 tx=0).stop 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 msgr2=0x7f2a380cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a2cff9640 1 --2- 172.21.15.32:0/3211152968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2a38159810 0x7f2a380cbc80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a30098130 tx=0x7f2a24005800 comp rx=0 tx=0).stop 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.756+0000 7f2a3eba3640 1 -- 172.21.15.32:0/3211152968 reap_dead start 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.760+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 shutdown_connections 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.760+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 >> 172.21.15.32:0/3211152968 conn(0x7f2a380cc7a0 msgr2=0x7f2a38151290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.760+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 shutdown_connections 2024-03-16T20:11:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.760+0000 7f2a2cff9640 1 -- 172.21.15.32:0/3211152968 wait complete. 2024-03-16T20:11:45.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-16T20:11:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 Processor -- start 2024-03-16T20:11:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 -- start start 2024-03-16T20:11:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 0x7f4644157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 0x7f46441519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464406fd00 con 0x7f46440c5f40 2024-03-16T20:11:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4644073550 con 0x7f46440c6320 2024-03-16T20:11:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.888+0000 7f464be04640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644070b00 con 0x7f4644158390 2024-03-16T20:11:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.892+0000 7f464939f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 0x7f4644157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.892+0000 7f464a3a1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 0x7f46441519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.892+0000 7f464939f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 0x7f4644157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58192/0 (socket says 172.21.15.32:58192) 2024-03-16T20:11:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.892+0000 7f464939f640 1 -- 172.21.15.32:0/2273890047 learned_addr learned my addr 172.21.15.32:0/2273890047 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.896+0000 7f464a3a1640 1 -- 172.21.15.32:0/2273890047 >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 msgr2=0x7f4644157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.896+0000 7f464a3a1640 1 --2- 172.21.15.32:0/2273890047 >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 0x7f4644157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.896+0000 7f464a3a1640 1 -- 172.21.15.32:0/2273890047 >> v1:172.21.15.32:6789/0 conn(0x7f46440c5f40 legacy=0x7f4644072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.896+0000 7f464a3a1640 1 -- 172.21.15.32:0/2273890047 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644158820 con 0x7f4644158390 2024-03-16T20:11:45.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.900+0000 7f464a3a1640 1 --2- 172.21.15.32:0/2273890047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 0x7f46441519f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f463c009d90 tx=0x7f463c01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77d098686086460 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f4648b9e640 1 -- 172.21.15.32:0/2273890047 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a57a0 con 0x7f4644158390 2024-03-16T20:11:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f4648b9e640 1 -- 172.21.15.32:0/2273890047 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0a5900 con 0x7f4644158390 2024-03-16T20:11:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f4648b9e640 1 -- 172.21.15.32:0/2273890047 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a5c90 con 0x7f4644158390 2024-03-16T20:11:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- 172.21.15.32:0/2273890047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 msgr2=0x7f46441519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 --2- 172.21.15.32:0/2273890047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 0x7f46441519f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f463c009d90 tx=0x7f463c01c8e0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- 172.21.15.32:0/2273890047 shutdown_connections 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 --2- 172.21.15.32:0/2273890047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4644158390 0x7f46441519f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 --2- 172.21.15.32:0/2273890047 >> v2:172.21.15.32:3300/0 conn(0x7f46440c6320 0x7f4644157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- 172.21.15.32:0/2273890047 >> 172.21.15.32:0/2273890047 conn(0x7f46440cc7a0 msgr2=0x7f46440ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- 172.21.15.32:0/2273890047 shutdown_connections 2024-03-16T20:11:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- 172.21.15.32:0/2273890047 wait complete. 2024-03-16T20:11:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 Processor -- start 2024-03-16T20:11:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.908+0000 7f464be04640 1 -- start start 2024-03-16T20:11:45.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46440c5f40 0x7f46441f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46440c6320 0x7f46441488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464414fff0 con 0x7f4644158390 2024-03-16T20:11:45.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f28c0 con 0x7f46440c5f40 2024-03-16T20:11:45.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464be04640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f3440 con 0x7f46440c6320 2024-03-16T20:11:45.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f4649ba0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46440c5f40 0x7f46441f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464a3a1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464a3a1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58198/0 (socket says 172.21.15.32:58198) 2024-03-16T20:11:45.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.912+0000 7f464a3a1640 1 -- 172.21.15.32:0/2288964851 learned_addr learned my addr 172.21.15.32:0/2288964851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:45.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.916+0000 7f464939f640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46440c6320 0x7f46441488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.920+0000 7f464a3a1640 1 -- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46440c6320 msgr2=0x7f46441488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.920+0000 7f464a3a1640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46440c6320 0x7f46441488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.920+0000 7f464a3a1640 1 -- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46440c5f40 msgr2=0x7f46441f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.920+0000 7f464a3a1640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46440c5f40 0x7f46441f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.920+0000 7f464a3a1640 1 -- 172.21.15.32:0/2288964851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009a70 con 0x7f4644158390 2024-03-16T20:11:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.924+0000 7f464a3a1640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b0040 tx=0x7f463c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b9650 con 0x7f4644158390 2024-03-16T20:11:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0b97b0 con 0x7f4644158390 2024-03-16T20:11:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b9b60 con 0x7f4644158390 2024-03-16T20:11:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46441f4c70 con 0x7f4644158390 2024-03-16T20:11:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441f2df0 con 0x7f4644158390 2024-03-16T20:11:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.928+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644068000 con 0x7f4644158390 2024-03-16T20:11:45.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.964+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f463c0b9cc0 con 0x7f4644158390 2024-03-16T20:11:45.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.964+0000 7f463a7fc640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 0x7f4628069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.964+0000 7f4649ba0640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 0x7f4628069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.964+0000 7f4649ba0640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 0x7f4628069380 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4640001770 tx=0x7f4640009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:45.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:45.972+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f463c0f0c40 con 0x7f4644158390 2024-03-16T20:11:46.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.284+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f463c0ecee0 con 0x7f4644158390 2024-03-16T20:11:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.452+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f464406bba0 con 0x7f4628066ec0 2024-03-16T20:11:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.452+0000 7f463a7fc640 1 -- 172.21.15.32:0/2288964851 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46400013b0 con 0x7f4628066ec0 2024-03-16T20:11:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 4 to deep-scrub 2024-03-16T20:11:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 msgr2=0x7f4628069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 0x7f4628069380 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4640001770 tx=0x7f4640009040 comp rx=0 tx=0).stop 2024-03-16T20:11:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 msgr2=0x7f464414fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b0040 tx=0x7f463c019680 comp rx=0 tx=0).stop 2024-03-16T20:11:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 shutdown_connections 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f4628066ec0 0x7f4628069380 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644158390 0x7f464414fab0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46440c6320 0x7f46441488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 --2- 172.21.15.32:0/2288964851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46440c5f40 0x7f46441f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 >> 172.21.15.32:0/2288964851 conn(0x7f46440cc7a0 msgr2=0x7f46440c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 shutdown_connections 2024-03-16T20:11:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.456+0000 7f464be04640 1 -- 172.21.15.32:0/2288964851 wait complete. 2024-03-16T20:11:46.464 INFO:tasks.ceph:Scrubbing osd.5 2024-03-16T20:11:46.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.576+0000 7f74f4fc8640 1 Processor -- start 2024-03-16T20:11:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.576+0000 7f74f4fc8640 1 -- start start 2024-03-16T20:11:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74f4fc8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f74f0150de0 0x7f74f00c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74f4fc8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f00c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74f4fc8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f0072850 con 0x7f74f015a140 2024-03-16T20:11:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74f4fc8640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f00c8e50 con 0x7f74f0150de0 2024-03-16T20:11:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74f4fc8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f015a560 con 0x7f74f00c8080 2024-03-16T20:11:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74ee59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f74f0150de0 0x7f74f00c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74eed9d640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f00c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74ee59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f74f0150de0 0x7f74f00c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.32:58684/0 (socket says 172.21.15.32:58684) 2024-03-16T20:11:46.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.580+0000 7f74ee59c640 1 -- 172.21.15.32:0/2970317531 learned_addr learned my addr 172.21.15.32:0/2970317531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.584+0000 7f74ee59c640 1 -- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 msgr2=0x7f74f00c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.584+0000 7f74ee59c640 1 --2- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f00c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.584+0000 7f74ee59c640 1 -- 172.21.15.32:0/2970317531 >> v1:172.21.15.59:6789/0 conn(0x7f74f015a140 legacy=0x7f74f00c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:46.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.584+0000 7f74ee59c640 1 -- 172.21.15.32:0/2970317531 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f00c8510 con 0x7f74f0150de0 2024-03-16T20:11:46.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.592+0000 7f74ee59c640 1 --2- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00c6fd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009870 tx=0x7f74e001c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bd7e83be84129bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74edd9b640 1 -- 172.21.15.32:0/2970317531 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e00ab600 con 0x7f74f0150de0 2024-03-16T20:11:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74edd9b640 1 -- 172.21.15.32:0/2970317531 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e00a7050 con 0x7f74f0150de0 2024-03-16T20:11:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74edd9b640 1 -- 172.21.15.32:0/2970317531 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e00ab980 con 0x7f74f0150de0 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 msgr2=0x7f74f00c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00c6fd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009870 tx=0x7f74e001c490 comp rx=0 tx=0).stop 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- 172.21.15.32:0/2970317531 shutdown_connections 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f00c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/2970317531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00c6fd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- 172.21.15.32:0/2970317531 >> 172.21.15.32:0/2970317531 conn(0x7f74f00cc7a0 msgr2=0x7f74f00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- 172.21.15.32:0/2970317531 shutdown_connections 2024-03-16T20:11:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- 172.21.15.32:0/2970317531 wait complete. 2024-03-16T20:11:46.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 Processor -- start 2024-03-16T20:11:46.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.600+0000 7f74f4fc8640 1 -- start start 2024-03-16T20:11:46.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f014bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 0x7f74f014e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00cc210 con 0x7f74f015a140 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f014c130 con 0x7f74f0150de0 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74f4fc8640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f014fc30 con 0x7f74f00c8080 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74eed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 0x7f74f014e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74eed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 0x7f74f014e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58208/0 (socket says 172.21.15.32:58208) 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74eed9d640 1 -- 172.21.15.32:0/3647996549 learned_addr learned my addr 172.21.15.32:0/3647996549 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74ee59c640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f014bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.604+0000 7f74e7fff640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.608+0000 7f74eed9d640 1 -- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 msgr2=0x7f74f014bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.608+0000 7f74eed9d640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74f00c8080 0x7f74f014bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.608+0000 7f74eed9d640 1 -- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 msgr2=0x7f74f00cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.608+0000 7f74eed9d640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74f0150de0 0x7f74f00cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.608+0000 7f74eed9d640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00094d0 con 0x7f74f015a140 2024-03-16T20:11:46.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.612+0000 7f74eed9d640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 0x7f74f014e210 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009d30 tx=0x7f74e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8009ba0 con 0x7f74f015a140 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e800f040 con 0x7f74f015a140 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f0150020 con 0x7f74f015a140 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e80146c0 con 0x7f74f015a140 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f014b110 con 0x7f74f015a140 2024-03-16T20:11:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.616+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74f00c7bf0 con 0x7f74f015a140 2024-03-16T20:11:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.640+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f74e8014820 con 0x7f74f015a140 2024-03-16T20:11:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.640+0000 7f74e6ffd640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f74d0066e40 0x7f74d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.640+0000 7f74ee59c640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f74d0066e40 0x7f74d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.644+0000 7f74ee59c640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f74d0066e40 0x7f74d0069300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f74e00ad000 tx=0x7f74e00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f74e8056bf0 con 0x7f74f015a140 2024-03-16T20:11:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e6ffd640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f74d0070860 0x7f74d0072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74d00733d0 con 0x7f74d0070860 2024-03-16T20:11:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74e800ced0 con 0x7f74f015a140 2024-03-16T20:11:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e7fff640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f74d0070860 0x7f74d0072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.648+0000 7f74e7fff640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f74d0070860 0x7f74d0072d00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.652+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== osd.5 v2:172.21.15.59:6816/2401239785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f74d00733d0 con 0x7f74d0070860 2024-03-16T20:11:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 --> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f74f00c7910 con 0x7f74d0070860 2024-03-16T20:11:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74e6ffd640 1 -- 172.21.15.32:0/3647996549 <== osd.5 v2:172.21.15.59:6816/2401239785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f74d8001490 con 0x7f74d0070860 2024-03-16T20:11:46.657 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:46.657 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:46.657 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f74d0070860 msgr2=0x7f74d0072d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.59:6816/2401239785,v1:172.21.15.59:6817/2401239785] conn(0x7f74d0070860 0x7f74d0072d00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f74d0066e40 msgr2=0x7f74d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f74d0066e40 0x7f74d0069300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f74e00ad000 tx=0x7f74e00a6000 comp rx=0 tx=0).stop 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 msgr2=0x7f74f014e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 --2- 172.21.15.32:0/3647996549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74f015a140 0x7f74f014e210 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009d30 tx=0x7f74e800c490 comp rx=0 tx=0).stop 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74eed9d640 1 -- 172.21.15.32:0/3647996549 reap_dead start 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 shutdown_connections 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 >> 172.21.15.32:0/3647996549 conn(0x7f74f00cc7a0 msgr2=0x7f74f014d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 shutdown_connections 2024-03-16T20:11:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.664+0000 7f74f4fc8640 1 -- 172.21.15.32:0/3647996549 wait complete. 2024-03-16T20:11:46.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-16T20:11:46.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 Processor -- start 2024-03-16T20:11:46.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 -- start start 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe468072bf0 0x7fe4680c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072fd0 0x7fe4680c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680c8240 con 0x7fe46815a770 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4680c3ad0 con 0x7fe468072bf0 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe46e035640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680c8cd0 con 0x7fe468072fd0 2024-03-16T20:11:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe467fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe46815a770 0x7fe46815ac00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.32:54666/0 (socket says 172.21.15.32:54666) 2024-03-16T20:11:46.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.796+0000 7fe467fff640 1 -- 172.21.15.32:0/3203626263 learned_addr learned my addr 172.21.15.32:0/3203626263 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:11:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.804+0000 7fe4677fe640 1 --2- 172.21.15.32:0/3203626263 >> v2:172.21.15.32:3300/0 conn(0x7fe468072bf0 0x7fe4680c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.808+0000 7fe4677fe640 1 -- 172.21.15.32:0/3203626263 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072fd0 msgr2=0x7fe4680c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.808+0000 7fe4677fe640 1 --2- 172.21.15.32:0/3203626263 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072fd0 0x7fe4680c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.808+0000 7fe4677fe640 1 -- 172.21.15.32:0/3203626263 >> v1:172.21.15.59:6789/0 conn(0x7fe46815a770 legacy=0x7fe46815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.808+0000 7fe4677fe640 1 -- 172.21.15.32:0/3203626263 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4680c3d20 con 0x7fe468072bf0 2024-03-16T20:11:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.812+0000 7fe4677fe640 1 --2- 172.21.15.32:0/3203626263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072bf0 0x7fe4680c7830 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe4580097c0 tx=0x7fe45801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442e98eb8dba031c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe4667fc640 1 -- 172.21.15.32:0/3203626263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4580a8990 con 0x7fe468072bf0 2024-03-16T20:11:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe4667fc640 1 -- 172.21.15.32:0/3203626263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4580a8af0 con 0x7fe468072bf0 2024-03-16T20:11:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe4667fc640 1 -- 172.21.15.32:0/3203626263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4580a8e80 con 0x7fe468072bf0 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- 172.21.15.32:0/3203626263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072bf0 msgr2=0x7fe4680c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 --2- 172.21.15.32:0/3203626263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072bf0 0x7fe4680c7830 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe4580097c0 tx=0x7fe45801ec90 comp rx=0 tx=0).stop 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- 172.21.15.32:0/3203626263 shutdown_connections 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 --2- 172.21.15.32:0/3203626263 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072fd0 0x7fe4680c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 --2- 172.21.15.32:0/3203626263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072bf0 0x7fe4680c7830 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- 172.21.15.32:0/3203626263 >> 172.21.15.32:0/3203626263 conn(0x7fe4680c92d0 msgr2=0x7fe4680c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- 172.21.15.32:0/3203626263 shutdown_connections 2024-03-16T20:11:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- 172.21.15.32:0/3203626263 wait complete. 2024-03-16T20:11:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 Processor -- start 2024-03-16T20:11:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.820+0000 7fe46e035640 1 -- start start 2024-03-16T20:11:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 0x7fe4681421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072fd0 0x7fe4680cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 0x7fe4680cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46815a190 con 0x7fe468072fd0 2024-03-16T20:11:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46814ef40 con 0x7fe46815a770 2024-03-16T20:11:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe46e035640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468154f00 con 0x7fe468072bf0 2024-03-16T20:11:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe4677fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 0x7fe4681421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe467fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 0x7fe4680cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe4677fe640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 0x7fe4681421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:44958/0 (socket says 172.21.15.32:44958) 2024-03-16T20:11:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe4677fe640 1 -- 172.21.15.32:0/767398998 learned_addr learned my addr 172.21.15.32:0/767398998 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.824+0000 7fe466ffd640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072fd0 0x7fe4680cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.828+0000 7fe467fff640 1 -- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 msgr2=0x7fe4681421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.828+0000 7fe467fff640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 0x7fe4681421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.828+0000 7fe467fff640 1 -- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072fd0 msgr2=0x7fe4680cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.828+0000 7fe467fff640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072fd0 0x7fe4680cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.828+0000 7fe467fff640 1 -- 172.21.15.32:0/767398998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe458009420 con 0x7fe46815a770 2024-03-16T20:11:46.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.832+0000 7fe467fff640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 0x7fe4680cbfa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe46000ca70 tx=0x7fe46000cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000aa90 con 0x7fe46815a770 2024-03-16T20:11:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468159bd0 con 0x7fe46815a770 2024-03-16T20:11:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46814e8f0 con 0x7fe46815a770 2024-03-16T20:11:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468068000 con 0x7fe46815a770 2024-03-16T20:11:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460001a30 con 0x7fe46815a770 2024-03-16T20:11:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:46.836+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000ae30 con 0x7fe46815a770 2024-03-16T20:11:47.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.136+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe460012400 con 0x7fe46815a770 2024-03-16T20:11:47.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.136+0000 7fe44ffff640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 0x7fe454069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.140+0000 7fe4677fe640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 0x7fe454069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.140+0000 7fe4677fe640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 0x7fe454069300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe45801d340 tx=0x7fe458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.144+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fe4600557e0 con 0x7fe46815a770 2024-03-16T20:11:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.168+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7fe460051a80 con 0x7fe46815a770 2024-03-16T20:11:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.344+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4680707b0 con 0x7fe454066e40 2024-03-16T20:11:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.344+0000 7fe44ffff640 1 -- 172.21.15.32:0/767398998 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe4680707b0 con 0x7fe454066e40 2024-03-16T20:11:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 5 to deep-scrub 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 msgr2=0x7fe454069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 0x7fe454069300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe45801d340 tx=0x7fe458008040 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 msgr2=0x7fe4680cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 0x7fe4680cbfa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe46000ca70 tx=0x7fe46000cf30 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 shutdown_connections 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fe454066e40 0x7fe454069300 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe46815a770 0x7fe4680cbfa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468072fd0 0x7fe4680cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 --2- 172.21.15.32:0/767398998 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe468072bf0 0x7fe4681421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 >> 172.21.15.32:0/767398998 conn(0x7fe4680c92d0 msgr2=0x7fe46806f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 shutdown_connections 2024-03-16T20:11:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.348+0000 7fe46e035640 1 -- 172.21.15.32:0/767398998 wait complete. 2024-03-16T20:11:47.363 INFO:tasks.ceph:Scrubbing osd.6 2024-03-16T20:11:47.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 Processor -- start 2024-03-16T20:11:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 -- start start 2024-03-16T20:11:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5880c5f40 0x7fc58806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc588157bf0 0x7fc588158060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc588073610 con 0x7fc5880c6320 2024-03-16T20:11:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5880c7d70 con 0x7fc5880c5f40 2024-03-16T20:11:47.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc58daed640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58806b500 con 0x7fc588157bf0 2024-03-16T20:11:47.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc5877fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5880c5f40 0x7fc58806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.484+0000 7fc587fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc588157bf0 0x7fc588158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5880c5f40 0x7fc58806afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58242/0 (socket says 172.21.15.32:58242) 2024-03-16T20:11:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 -- 172.21.15.32:0/3899865860 learned_addr learned my addr 172.21.15.32:0/3899865860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 -- 172.21.15.32:0/3899865860 >> v2:172.21.15.59:3300/0 conn(0x7fc588157bf0 msgr2=0x7fc588158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 --2- 172.21.15.32:0/3899865860 >> v2:172.21.15.59:3300/0 conn(0x7fc588157bf0 0x7fc588158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 -- 172.21.15.32:0/3899865860 >> v1:172.21.15.32:6789/0 conn(0x7fc5880c6320 legacy=0x7fc5880c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.488+0000 7fc5877fe640 1 -- 172.21.15.32:0/3899865860 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880c6fd0 con 0x7fc5880c5f40 2024-03-16T20:11:47.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.496+0000 7fc5877fe640 1 --2- 172.21.15.32:0/3899865860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58806afc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc5880c71c0 tx=0x7fc57801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6999aabe28f8236 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.500+0000 7fc5867fc640 1 -- 172.21.15.32:0/3899865860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5780a8a10 con 0x7fc5880c5f40 2024-03-16T20:11:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.500+0000 7fc5867fc640 1 -- 172.21.15.32:0/3899865860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5780a8b70 con 0x7fc5880c5f40 2024-03-16T20:11:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc5867fc640 1 -- 172.21.15.32:0/3899865860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5780b0440 con 0x7fc5880c5f40 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- 172.21.15.32:0/3899865860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 msgr2=0x7fc58806afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 --2- 172.21.15.32:0/3899865860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58806afc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc5880c71c0 tx=0x7fc57801ed10 comp rx=0 tx=0).stop 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- 172.21.15.32:0/3899865860 shutdown_connections 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 --2- 172.21.15.32:0/3899865860 >> v2:172.21.15.59:3300/0 conn(0x7fc588157bf0 0x7fc588158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 --2- 172.21.15.32:0/3899865860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58806afc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- 172.21.15.32:0/3899865860 >> 172.21.15.32:0/3899865860 conn(0x7fc5880cc7a0 msgr2=0x7fc5880ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- 172.21.15.32:0/3899865860 shutdown_connections 2024-03-16T20:11:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- 172.21.15.32:0/3899865860 wait complete. 2024-03-16T20:11:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 Processor -- start 2024-03-16T20:11:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.504+0000 7fc58daed640 1 -- start start 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58815b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5880c6320 0x7fc588146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc588157bf0 0x7fc588153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5881592d0 con 0x7fc5880c5f40 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588159e50 con 0x7fc5880c6320 2024-03-16T20:11:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc58daed640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58815b450 con 0x7fc588157bf0 2024-03-16T20:11:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58815b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58815b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58256/0 (socket says 172.21.15.32:58256) 2024-03-16T20:11:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc5877fe640 1 -- 172.21.15.32:0/229044435 learned_addr learned my addr 172.21.15.32:0/229044435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc587fff640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc588157bf0 0x7fc588153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.508+0000 7fc586ffd640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5880c6320 0x7fc588146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.512+0000 7fc5877fe640 1 -- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc588157bf0 msgr2=0x7fc588153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.512+0000 7fc5877fe640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc588157bf0 0x7fc588153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.512+0000 7fc5877fe640 1 -- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5880c6320 msgr2=0x7fc588146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.512+0000 7fc5877fe640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5880c6320 0x7fc588146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.512+0000 7fc5877fe640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5780094d0 con 0x7fc5880c5f40 2024-03-16T20:11:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.516+0000 7fc5877fe640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58815b870 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc588070d80 tx=0x7fc57801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5780b1c60 con 0x7fc5880c5f40 2024-03-16T20:11:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5780b1dc0 con 0x7fc5880c5f40 2024-03-16T20:11:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58815a2e0 con 0x7fc5880c5f40 2024-03-16T20:11:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5780b9640 con 0x7fc5880c5f40 2024-03-16T20:11:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc588159520 con 0x7fc5880c5f40 2024-03-16T20:11:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.536+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5880cb850 con 0x7fc5880c5f40 2024-03-16T20:11:47.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.568+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc5780b7070 con 0x7fc5880c5f40 2024-03-16T20:11:47.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.568+0000 7fc56ffff640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc574066e40 0x7fc574069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.568+0000 7fc586ffd640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc574066e40 0x7fc574069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.572+0000 7fc586ffd640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc574066e40 0x7fc574069300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc5880c31e0 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.572+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc5780f37e0 con 0x7fc5880c5f40 2024-03-16T20:11:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.572+0000 7fc56ffff640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7fc574070860 0x7fc574072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.572+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5740733d0 con 0x7fc574070860 2024-03-16T20:11:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.576+0000 7fc587fff640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7fc574070860 0x7fc574072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.576+0000 7fc587fff640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7fc574070860 0x7fc574072d00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.576+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== osd.6 v2:172.21.15.59:6801/459431804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc5740733d0 con 0x7fc574070860 2024-03-16T20:11:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.576+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5780efa80 con 0x7fc5880c5f40 2024-03-16T20:11:47.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 --> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc588072f40 con 0x7fc574070860 2024-03-16T20:11:47.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc56ffff640 1 -- 172.21.15.32:0/229044435 <== osd.6 v2:172.21.15.59:6801/459431804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc588072f40 con 0x7fc574070860 2024-03-16T20:11:47.586 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:47.586 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7fc574070860 msgr2=0x7fc574072d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.59:6801/459431804,v1:172.21.15.59:6803/459431804] conn(0x7fc574070860 0x7fc574072d00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc574066e40 msgr2=0x7fc574069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fc574066e40 0x7fc574069300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc5880c31e0 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 msgr2=0x7fc58815b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 --2- 172.21.15.32:0/229044435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5880c5f40 0x7fc58815b870 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc588070d80 tx=0x7fc57801bab0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc587fff640 1 -- 172.21.15.32:0/229044435 reap_dead start 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 shutdown_connections 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 >> 172.21.15.32:0/229044435 conn(0x7fc5880cc7a0 msgr2=0x7fc5880c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:47.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 shutdown_connections 2024-03-16T20:11:47.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.592+0000 7fc58daed640 1 -- 172.21.15.32:0/229044435 wait complete. 2024-03-16T20:11:47.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-16T20:11:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 Processor -- start 2024-03-16T20:11:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 -- start start 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff790154200 0x7ff790072370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff7900c7d20 0x7ff7900728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790071930 0x7ff79015a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7901545e0 con 0x7ff790154200 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff790072eb0 con 0x7ff7900c7d20 2024-03-16T20:11:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.728+0000 7ff795c70640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff790159dd0 con 0x7ff790071930 2024-03-16T20:11:47.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.736+0000 7ff78f7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff790154200 0x7ff790072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.736+0000 7ff78f7fe640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff790154200 0x7ff790072370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58260/0 (socket says 172.21.15.32:58260) 2024-03-16T20:11:47.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.736+0000 7ff78f7fe640 1 -- 172.21.15.32:0/2556849514 learned_addr learned my addr 172.21.15.32:0/2556849514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:47.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.736+0000 7ff787fff640 1 --2- 172.21.15.32:0/2556849514 >> v2:172.21.15.59:3300/0 conn(0x7ff7900c7d20 0x7ff7900728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.740+0000 7ff78f7fe640 1 -- 172.21.15.32:0/2556849514 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790071930 msgr2=0x7ff79015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.740+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/2556849514 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790071930 0x7ff79015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.740+0000 7ff78f7fe640 1 -- 172.21.15.32:0/2556849514 >> v2:172.21.15.59:3300/0 conn(0x7ff7900c7d20 msgr2=0x7ff7900728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.740+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/2556849514 >> v2:172.21.15.59:3300/0 conn(0x7ff7900c7d20 0x7ff7900728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.740+0000 7ff78f7fe640 1 -- 172.21.15.32:0/2556849514 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900c6900 con 0x7ff790154200 2024-03-16T20:11:47.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.800+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/2556849514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790154200 0x7ff790072370 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff7900c15a0 tx=0x7ff78001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ef86b6412daee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.812+0000 7ff78effd640 1 -- 172.21.15.32:0/2556849514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7800ab620 con 0x7ff790154200 2024-03-16T20:11:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff78effd640 1 -- 172.21.15.32:0/2556849514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7800ab780 con 0x7ff790154200 2024-03-16T20:11:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff78effd640 1 -- 172.21.15.32:0/2556849514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7800abb10 con 0x7ff790154200 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 -- 172.21.15.32:0/2556849514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790154200 msgr2=0x7ff790072370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 --2- 172.21.15.32:0/2556849514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790154200 0x7ff790072370 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff7900c15a0 tx=0x7ff78001c460 comp rx=0 tx=0).stop 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 -- 172.21.15.32:0/2556849514 shutdown_connections 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 --2- 172.21.15.32:0/2556849514 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790071930 0x7ff79015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 --2- 172.21.15.32:0/2556849514 >> v2:172.21.15.59:3300/0 conn(0x7ff7900c7d20 0x7ff7900728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 --2- 172.21.15.32:0/2556849514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790154200 0x7ff790072370 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 -- 172.21.15.32:0/2556849514 >> 172.21.15.32:0/2556849514 conn(0x7ff79006f3d0 msgr2=0x7ff79006f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 -- 172.21.15.32:0/2556849514 shutdown_connections 2024-03-16T20:11:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.816+0000 7ff795c70640 1 -- 172.21.15.32:0/2556849514 wait complete. 2024-03-16T20:11:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 Processor -- start 2024-03-16T20:11:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 -- start start 2024-03-16T20:11:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7900c7d20 0x7ff79014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790154200 0x7ff7901409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79014e270 con 0x7ff790071930 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79014e830 con 0x7ff7900c7d20 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff795c70640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff790151870 con 0x7ff790154200 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff78f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff78f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58272/0 (socket says 172.21.15.32:58272) 2024-03-16T20:11:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.820+0000 7ff78f7fe640 1 -- 172.21.15.32:0/1638358775 learned_addr learned my addr 172.21.15.32:0/1638358775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.824+0000 7ff78f7fe640 1 -- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790154200 msgr2=0x7ff7901409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.824+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790154200 0x7ff7901409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.824+0000 7ff78f7fe640 1 -- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7900c7d20 msgr2=0x7ff79014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.824+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7900c7d20 0x7ff79014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.824+0000 7ff78f7fe640 1 -- 172.21.15.32:0/1638358775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7800094d0 con 0x7ff790071930 2024-03-16T20:11:47.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.828+0000 7ff78f7fe640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff78001ab10 tx=0x7ff78001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:47.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7800be3f0 con 0x7ff790071930 2024-03-16T20:11:47.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7800be570 con 0x7ff790071930 2024-03-16T20:11:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7800be920 con 0x7ff790071930 2024-03-16T20:11:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7901527a0 con 0x7ff790071930 2024-03-16T20:11:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff790152a40 con 0x7ff790071930 2024-03-16T20:11:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.832+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790068000 con 0x7ff790071930 2024-03-16T20:11:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.860+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff7800b43f0 con 0x7ff790071930 2024-03-16T20:11:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.860+0000 7ff78cff9640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 0x7ff770069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.860+0000 7ff787fff640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 0x7ff770069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:47.860+0000 7ff787fff640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 0x7ff770069300 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff79006b3a0 tx=0x7ff778008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.156+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7ff7800efce0 con 0x7ff790071930 2024-03-16T20:11:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.156+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7ff7800ebf80 con 0x7ff790071930 2024-03-16T20:11:48.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.400+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7900671f0 con 0x7ff770066e40 2024-03-16T20:11:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.400+0000 7ff78cff9640 1 -- 172.21.15.32:0/1638358775 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff7900671f0 con 0x7ff770066e40 2024-03-16T20:11:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 6 to deep-scrub 2024-03-16T20:11:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 msgr2=0x7ff770069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 0x7ff770069300 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff79006b3a0 tx=0x7ff778008040 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 msgr2=0x7ff790144cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff78001ab10 tx=0x7ff78001aca0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 shutdown_connections 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7ff770066e40 0x7ff770069300 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff790154200 0x7ff7901409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7900c7d20 0x7ff79014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 --2- 172.21.15.32:0/1638358775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff790071930 0x7ff790144cb0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 >> 172.21.15.32:0/1638358775 conn(0x7ff79006f3d0 msgr2=0x7ff7901479f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 shutdown_connections 2024-03-16T20:11:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.404+0000 7ff795c70640 1 -- 172.21.15.32:0/1638358775 wait complete. 2024-03-16T20:11:48.412 INFO:tasks.ceph:Scrubbing osd.7 2024-03-16T20:11:48.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-16T20:11:48.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.524+0000 7fbdf063e640 1 Processor -- start 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.524+0000 7fbdf063e640 1 -- start start 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbde81574d0 0x7fbde80cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbde815a8c0 0x7fbde806f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde815e3f0 con 0x7fbde815a8c0 2024-03-16T20:11:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde8159f30 con 0x7fbde81574d0 2024-03-16T20:11:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdf063e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde815ad20 con 0x7fbde80cdc00 2024-03-16T20:11:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdeebdb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbde815a8c0 0x7fbde806f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdeebdb640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbde815a8c0 0x7fbde806f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58274/0 (socket says 172.21.15.32:58274) 2024-03-16T20:11:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1635002339 learned_addr learned my addr 172.21.15.32:0/1635002339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdedbd9640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.528+0000 7fbdee3da640 1 --2- 172.21.15.32:0/1635002339 >> v2:172.21.15.59:3300/0 conn(0x7fbde81574d0 0x7fbde80cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.532+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1635002339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 msgr2=0x7fbde815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.532+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.532+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1635002339 >> v2:172.21.15.59:3300/0 conn(0x7fbde81574d0 msgr2=0x7fbde80cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.532+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1635002339 >> v2:172.21.15.59:3300/0 conn(0x7fbde81574d0 0x7fbde80cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.532+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1635002339 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde815d5b0 con 0x7fbde815a8c0 2024-03-16T20:11:48.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.536+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde806f2d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbde80c7e30 tx=0x7fbde001f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381eb87af149016b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbded3d8640 1 -- 172.21.15.32:0/1635002339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00af400 con 0x7fbde815a8c0 2024-03-16T20:11:48.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbded3d8640 1 -- 172.21.15.32:0/1635002339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00ab050 con 0x7fbde815a8c0 2024-03-16T20:11:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbded3d8640 1 -- 172.21.15.32:0/1635002339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00af7c0 con 0x7fbde815a8c0 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- 172.21.15.32:0/1635002339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 msgr2=0x7fbde806f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde806f2d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbde80c7e30 tx=0x7fbde001f200 comp rx=0 tx=0).stop 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- 172.21.15.32:0/1635002339 shutdown_connections 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde806f2d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1635002339 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1635002339 >> v2:172.21.15.59:3300/0 conn(0x7fbde81574d0 0x7fbde80cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- 172.21.15.32:0/1635002339 >> 172.21.15.32:0/1635002339 conn(0x7fbde806b5c0 msgr2=0x7fbde8066cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- 172.21.15.32:0/1635002339 shutdown_connections 2024-03-16T20:11:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- 172.21.15.32:0/1635002339 wait complete. 2024-03-16T20:11:48.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 Processor -- start 2024-03-16T20:11:48.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.540+0000 7fbdf063e640 1 -- start start 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde81e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde81574d0 0x7fbde815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde815b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80ccad0 con 0x7fbde815a8c0 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80cbbe0 con 0x7fbde81574d0 2024-03-16T20:11:48.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdf063e640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80cbec0 con 0x7fbde80cdc00 2024-03-16T20:11:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdeebdb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde815b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdeebdb640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde815b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58288/0 (socket says 172.21.15.32:58288) 2024-03-16T20:11:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1831210128 learned_addr learned my addr 172.21.15.32:0/1831210128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdee3da640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde81e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.544+0000 7fbdedbd9640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde81574d0 0x7fbde815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.548+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 msgr2=0x7fbde81e6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.548+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbde80cdc00 0x7fbde81e6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.548+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde81574d0 msgr2=0x7fbde815b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.548+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde81574d0 0x7fbde815b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.548+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0009b30 con 0x7fbde815a8c0 2024-03-16T20:11:48.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.552+0000 7fbdeebdb640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde815b810 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbde00b2040 tx=0x7fbde001bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.556+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00afd10 con 0x7fbde815a8c0 2024-03-16T20:11:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.556+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00afe90 con 0x7fbde815a8c0 2024-03-16T20:11:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.556+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde80cc1a0 con 0x7fbde815a8c0 2024-03-16T20:11:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.556+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00ba660 con 0x7fbde815a8c0 2024-03-16T20:11:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.556+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde80cc760 con 0x7fbde815a8c0 2024-03-16T20:11:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.560+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbde806a340 con 0x7fbde815a8c0 2024-03-16T20:11:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.596+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbde00b6040 con 0x7fbde815a8c0 2024-03-16T20:11:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.596+0000 7fbddeffd640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbdcc066e70 0x7fbdcc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.596+0000 7fbdee3da640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbdcc066e70 0x7fbdcc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.596+0000 7fbdee3da640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbdcc066e70 0x7fbdcc069330 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007a60 tx=0x7fbde4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.600+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fbde00f3e30 con 0x7fbde815a8c0 2024-03-16T20:11:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbddeffd640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7fbdcc0708a0 0x7fbdcc072d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdcc073410 con 0x7fbdcc0708a0 2024-03-16T20:11:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde00f00d0 con 0x7fbde815a8c0 2024-03-16T20:11:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbdedbd9640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7fbdcc0708a0 0x7fbdcc072d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbdedbd9640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7fbdcc0708a0 0x7fbdcc072d40 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.604+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== osd.7 v2:172.21.15.59:6800/1528567805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbdcc073410 con 0x7fbdcc0708a0 2024-03-16T20:11:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 --> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbde815de30 con 0x7fbdcc0708a0 2024-03-16T20:11:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbddeffd640 1 -- 172.21.15.32:0/1831210128 <== osd.7 v2:172.21.15.59:6800/1528567805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbdd8009810 con 0x7fbdcc0708a0 2024-03-16T20:11:48.620 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-16T20:11:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T20:11:48.621 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-16T20:11:48.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7fbdcc0708a0 msgr2=0x7fbdcc072d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.59:6800/1528567805,v1:172.21.15.59:6802/1528567805] conn(0x7fbdcc0708a0 0x7fbdcc072d40 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbdcc066e70 msgr2=0x7fbdcc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7fbdcc066e70 0x7fbdcc069330 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007a60 tx=0x7fbde4008040 comp rx=0 tx=0).stop 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 msgr2=0x7fbde815b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 --2- 172.21.15.32:0/1831210128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbde815a8c0 0x7fbde815b810 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbde00b2040 tx=0x7fbde001bfa0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdeebdb640 1 -- 172.21.15.32:0/1831210128 reap_dead start 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 shutdown_connections 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 >> 172.21.15.32:0/1831210128 conn(0x7fbde806b5c0 msgr2=0x7fbde8158dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 shutdown_connections 2024-03-16T20:11:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.628+0000 7fbdf063e640 1 -- 172.21.15.32:0/1831210128 wait complete. 2024-03-16T20:11:48.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-16T20:11:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 Processor -- start 2024-03-16T20:11:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 -- start start 2024-03-16T20:11:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 0x7f04c0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04c00c6320 0x7f04c0157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c006fd00 con 0x7f04c0158390 2024-03-16T20:11:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c0070b00 con 0x7f04c00c6320 2024-03-16T20:11:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.756+0000 7f04c6531640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c0158820 con 0x7f04c00c5f40 2024-03-16T20:11:48.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.764+0000 7f04bffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 0x7f04c0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.768+0000 7f04bf7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04c00c6320 0x7f04c0157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.768+0000 7f04bffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 0x7f04c0072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:45026/0 (socket says 172.21.15.32:45026) 2024-03-16T20:11:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.768+0000 7f04bffff640 1 -- 172.21.15.32:0/3968850447 learned_addr learned my addr 172.21.15.32:0/3968850447 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.784+0000 7f04bf7fe640 1 -- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 msgr2=0x7f04c0072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.784+0000 7f04bf7fe640 1 --2- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 0x7f04c0072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.784+0000 7f04bf7fe640 1 -- 172.21.15.32:0/3968850447 >> v1:172.21.15.59:6789/0 conn(0x7f04c0158390 legacy=0x7f04c01519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.784+0000 7f04bf7fe640 1 -- 172.21.15.32:0/3968850447 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c006ff30 con 0x7f04c00c6320 2024-03-16T20:11:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.792+0000 7f04bf7fe640 1 --2- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c6320 0x7f04c0157e50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097c0 tx=0x7f04b401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e7708b457ef467 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04beffd640 1 -- 172.21.15.32:0/3968850447 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab600 con 0x7f04c00c6320 2024-03-16T20:11:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04beffd640 1 -- 172.21.15.32:0/3968850447 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40a7050 con 0x7f04c00c6320 2024-03-16T20:11:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04beffd640 1 -- 172.21.15.32:0/3968850447 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab980 con 0x7f04c00c6320 2024-03-16T20:11:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c6320 msgr2=0x7f04c0157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c6320 0x7f04c0157e50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097c0 tx=0x7f04b401c410 comp rx=0 tx=0).stop 2024-03-16T20:11:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- 172.21.15.32:0/3968850447 shutdown_connections 2024-03-16T20:11:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c6320 0x7f04c0157e50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- 172.21.15.32:0/3968850447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c00c5f40 0x7f04c0072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- 172.21.15.32:0/3968850447 >> 172.21.15.32:0/3968850447 conn(0x7f04c00cc7a0 msgr2=0x7f04c00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- 172.21.15.32:0/3968850447 shutdown_connections 2024-03-16T20:11:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- 172.21.15.32:0/3968850447 wait complete. 2024-03-16T20:11:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 Processor -- start 2024-03-16T20:11:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- start start 2024-03-16T20:11:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c5f40 0x7f04c01450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c00c6320 0x7f04c0150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c014a350 con 0x7f04c00c6320 2024-03-16T20:11:48.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01f0630 con 0x7f04c00c5f40 2024-03-16T20:11:48.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.892+0000 7f04c6531640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01f08d0 con 0x7f04c0158390 2024-03-16T20:11:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.896+0000 7f04c4ace640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.896+0000 7f04c4ace640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:45042/0 (socket says 172.21.15.32:45042) 2024-03-16T20:11:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.896+0000 7f04c4ace640 1 -- 172.21.15.32:0/939906574 learned_addr learned my addr 172.21.15.32:0/939906574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:48.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.896+0000 7f04bffff640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c5f40 0x7f04c01450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.896+0000 7f04bf7fe640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c00c6320 0x7f04c0150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.900+0000 7f04c4ace640 1 -- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c5f40 msgr2=0x7f04c01450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.900+0000 7f04c4ace640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c5f40 0x7f04c01450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.900+0000 7f04c4ace640 1 -- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c00c6320 msgr2=0x7f04c0150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.900+0000 7f04c4ace640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c00c6320 0x7f04c0150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.900+0000 7f04c4ace640 1 -- 172.21.15.32:0/939906574 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009420 con 0x7f04c0158390 2024-03-16T20:11:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.904+0000 7f04c4ace640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04b800c730 tx=0x7f04b800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8003b80 con 0x7f04c0158390 2024-03-16T20:11:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b8003ce0 con 0x7f04c0158390 2024-03-16T20:11:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c01f0bb0 con 0x7f04c0158390 2024-03-16T20:11:48.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800aa30 con 0x7f04c0158390 2024-03-16T20:11:48.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c01f1170 con 0x7f04c0158390 2024-03-16T20:11:48.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.912+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c01e9bb0 con 0x7f04c0158390 2024-03-16T20:11:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.940+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f04b800ab90 con 0x7f04c0158390 2024-03-16T20:11:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.940+0000 7f04bcff9640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 0x7f04ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.940+0000 7f04bffff640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 0x7f04ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.940+0000 7f04bffff640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 0x7f04ac069300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009b50 tx=0x7f04b0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:48.944+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f04b800ae40 con 0x7f04c0158390 2024-03-16T20:11:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.232+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f04b8051eb0 con 0x7f04c0158390 2024-03-16T20:11:49.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.416+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c00c8aa0 con 0x7f04ac066e40 2024-03-16T20:11:49.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.416+0000 7f04bcff9640 1 -- 172.21.15.32:0/939906574 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04b0011ac0 con 0x7f04ac066e40 2024-03-16T20:11:49.410 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 7 to deep-scrub 2024-03-16T20:11:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 msgr2=0x7f04ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 0x7f04ac069300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009b50 tx=0x7f04b0009240 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 msgr2=0x7f04c014aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04b800c730 tx=0x7f04b800cbf0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 shutdown_connections 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f04ac066e40 0x7f04ac069300 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f04c0158390 0x7f04c014aa50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c00c6320 0x7f04c0150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 --2- 172.21.15.32:0/939906574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c00c5f40 0x7f04c01450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 >> 172.21.15.32:0/939906574 conn(0x7f04c00cc7a0 msgr2=0x7f04c00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 shutdown_connections 2024-03-16T20:11:49.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.420+0000 7f04c6531640 1 -- 172.21.15.32:0/939906574 wait complete. 2024-03-16T20:11:49.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:11:49.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.540+0000 7efcc61c7640 1 Processor -- start 2024-03-16T20:11:49.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.540+0000 7efcc61c7640 1 -- start start 2024-03-16T20:11:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcc61c7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcc00725e0 0x7efcc0073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcc61c7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcc61c7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc0151020 con 0x7efcc0150c40 2024-03-16T20:11:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcc61c7640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc015a000 con 0x7efcc00725e0 2024-03-16T20:11:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcc61c7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc015add0 con 0x7efcc015a420 2024-03-16T20:11:49.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcbeffd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcc00725e0 0x7efcc0073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcbffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcbffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.32:56442/0 (socket says 172.21.15.32:56442) 2024-03-16T20:11:49.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.544+0000 7efcbffff640 1 -- 172.21.15.32:0/230784437 learned_addr learned my addr 172.21.15.32:0/230784437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.548+0000 7efcbeffd640 1 -- 172.21.15.32:0/230784437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 msgr2=0x7efcc015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:11:49.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.548+0000 7efcbeffd640 1 --2- 172.21.15.32:0/230784437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.548+0000 7efcbeffd640 1 -- 172.21.15.32:0/230784437 >> v1:172.21.15.59:6789/0 conn(0x7efcc0150c40 legacy=0x7efcc0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.548+0000 7efcbeffd640 1 -- 172.21.15.32:0/230784437 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0072da0 con 0x7efcc00725e0 2024-03-16T20:11:49.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.552+0000 7efcbeffd640 1 --2- 172.21.15.32:0/230784437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00725e0 0x7efcc0073070 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efcb40097c0 tx=0x7efcb401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b6c270205440d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:49.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcbe7fc640 1 -- 172.21.15.32:0/230784437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40a6040 con 0x7efcc00725e0 2024-03-16T20:11:49.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcbe7fc640 1 -- 172.21.15.32:0/230784437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb40aa600 con 0x7efcc00725e0 2024-03-16T20:11:49.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcbe7fc640 1 -- 172.21.15.32:0/230784437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40aa990 con 0x7efcc00725e0 2024-03-16T20:11:49.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 -- 172.21.15.32:0/230784437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00725e0 msgr2=0x7efcc0073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 --2- 172.21.15.32:0/230784437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00725e0 0x7efcc0073070 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efcb40097c0 tx=0x7efcb401c4a0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 -- 172.21.15.32:0/230784437 shutdown_connections 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 --2- 172.21.15.32:0/230784437 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 --2- 172.21.15.32:0/230784437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00725e0 0x7efcc0073070 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 -- 172.21.15.32:0/230784437 >> 172.21.15.32:0/230784437 conn(0x7efcc006c0d0 msgr2=0x7efcc006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 -- 172.21.15.32:0/230784437 shutdown_connections 2024-03-16T20:11:49.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.556+0000 7efcc61c7640 1 -- 172.21.15.32:0/230784437 wait complete. 2024-03-16T20:11:49.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 Processor -- start 2024-03-16T20:11:49.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 -- start start 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcc00725e0 0x7efcc00cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc015cea0 con 0x7efcc0150c40 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00cbf10 con 0x7efcc00725e0 2024-03-16T20:11:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcc61c7640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00cb380 con 0x7efcc015a420 2024-03-16T20:11:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcbffff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50470/0 (socket says 172.21.15.32:50470) 2024-03-16T20:11:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcbeffd640 1 -- 172.21.15.32:0/2173766122 learned_addr learned my addr 172.21.15.32:0/2173766122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:11:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.560+0000 7efcbf7fe640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcc00725e0 0x7efcc00cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.564+0000 7efcbeffd640 1 -- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 msgr2=0x7efcc015b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.564+0000 7efcbeffd640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.564+0000 7efcbeffd640 1 -- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcc00725e0 msgr2=0x7efcc00cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.564+0000 7efcbeffd640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcc00725e0 0x7efcc00cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.564+0000 7efcbeffd640 1 -- 172.21.15.32:0/2173766122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4009420 con 0x7efcc0150c40 2024-03-16T20:11:49.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.568+0000 7efcbeffd640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efcb401a0d0 tx=0x7efcb401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40b3800 con 0x7efcc0150c40 2024-03-16T20:11:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb40b3980 con 0x7efcc0150c40 2024-03-16T20:11:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc015cac0 con 0x7efcc0150c40 2024-03-16T20:11:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40b3d30 con 0x7efcc0150c40 2024-03-16T20:11:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc00caae0 con 0x7efcc0150c40 2024-03-16T20:11:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.576+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc015be10 con 0x7efcc0150c40 2024-03-16T20:11:49.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.600+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efcb40c6430 con 0x7efcc0150c40 2024-03-16T20:11:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.600+0000 7efca7fff640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 0x7efcac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:11:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.600+0000 7efcbf7fe640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 0x7efcac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:11:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.600+0000 7efcbf7fe640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 0x7efcac069350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efcb0007fd0 tx=0x7efcb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:11:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.888+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efcb40f0520 con 0x7efcc0150c40 2024-03-16T20:11:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:49.892+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7efcb40ec7c0 con 0x7efcc0150c40 2024-03-16T20:11:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.096+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcc00c77f0 con 0x7efcac066e90 2024-03-16T20:11:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.104+0000 7efca7fff640 1 -- 172.21.15.32:0/2173766122 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+330922 (secure 0 0 0) 0x7efcc00c77f0 con 0x7efcac066e90 2024-03-16T20:11:50.101 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:11:50.103 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 msgr2=0x7efcac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 0x7efcac069350 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efcb0007fd0 tx=0x7efcb0009040 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 msgr2=0x7efcc015b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efcb401a0d0 tx=0x7efcb401ace0 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 shutdown_connections 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7efcac066e90 0x7efcac069350 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7efcc015a420 0x7efcc015b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0150c40 0x7efcc015b390 secure :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efcb401a0d0 tx=0x7efcb401ace0 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 --2- 172.21.15.32:0/2173766122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcc00725e0 0x7efcc00cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:11:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 >> 172.21.15.32:0/2173766122 conn(0x7efcc006c0d0 msgr2=0x7efcc014e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:11:50.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 shutdown_connections 2024-03-16T20:11:50.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:11:50.108+0000 7efcc61c7640 1 -- 172.21.15.32:0/2173766122 wait complete. 2024-03-16T20:11:50.121 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-03-16T20:11:48.849041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":334,"ondisk_log_size":334,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236848,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737872,"statfs":{"total":773094113280,"available":772851580928,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"2.1f","version":"32'20","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:42.724708+0000","last_change":"2024-03-16T20:11:42.724767+0000","last_active":"2024-03-16T20:11:42.724708+0000","last_peered":"2024-03-16T20:11:42.724708+0000","last_clean":"2024-03-16T20:11:42.724708+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:11:42.724708+0000","last_undegraded":"2024-03-16T20:11:42.724708+0000","last_fullsized":"2024-03-16T20:11:42.724708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_clean_scrub_stamp":"2024-03-16T20:11:42.724584+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:20.296590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.927855+0000","last_change":"2024-03-16T20:11:46.927899+0000","last_active":"2024-03-16T20:11:46.927855+0000","last_peered":"2024-03-16T20:11:46.927855+0000","last_clean":"2024-03-16T20:11:46.927855+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:11:46.927855+0000","last_undegraded":"2024-03-16T20:11:46.927855+0000","last_fullsized":"2024-03-16T20:11:46.927855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.927737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.399583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:18.385831+0000","last_change":"2024-03-16T19:56:43.494986+0000","last_active":"2024-03-16T20:11:18.385831+0000","last_peered":"2024-03-16T20:11:18.385831+0000","last_clean":"2024-03-16T20:11:18.385831+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:11:18.385831+0000","last_undegraded":"2024-03-16T20:11:18.385831+0000","last_fullsized":"2024-03-16T20:11:18.385831+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:33.487148+0000","last_change":"2024-03-16T19:56:33.963923+0000","last_active":"2024-03-16T20:10:33.487148+0000","last_peered":"2024-03-16T20:10:33.487148+0000","last_clean":"2024-03-16T20:10:33.487148+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:10:33.487148+0000","last_undegraded":"2024-03-16T20:10:33.487148+0000","last_fullsized":"2024-03-16T20:10:33.487148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.783367+0000","last_change":"2024-03-16T20:11:46.783404+0000","last_active":"2024-03-16T20:11:46.783367+0000","last_peered":"2024-03-16T20:11:46.783367+0000","last_clean":"2024-03-16T20:11:46.783367+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:11:46.783367+0000","last_undegraded":"2024-03-16T20:11:46.783367+0000","last_fullsized":"2024-03-16T20:11:46.783367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.783256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:53.479114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.479106+0000","last_change":"2024-03-16T19:56:43.457701+0000","last_active":"2024-03-16T20:08:49.479106+0000","last_peered":"2024-03-16T20:08:49.479106+0000","last_clean":"2024-03-16T20:08:49.479106+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:08:49.479106+0000","last_undegraded":"2024-03-16T20:08:49.479106+0000","last_fullsized":"2024-03-16T20:08:49.479106+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:28.131519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'13","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405198+0000","last_change":"2024-03-16T19:56:33.980347+0000","last_active":"2024-03-16T20:09:04.405198+0000","last_peered":"2024-03-16T20:09:04.405198+0000","last_clean":"2024-03-16T20:09:04.405198+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:09:04.405198+0000","last_undegraded":"2024-03-16T20:09:04.405198+0000","last_fullsized":"2024-03-16T20:09:04.405198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:48.466150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432089+0000","last_change":"2024-03-16T19:56:36.305357+0000","last_active":"2024-03-16T20:09:09.432089+0000","last_peered":"2024-03-16T20:09:09.432089+0000","last_clean":"2024-03-16T20:09:09.432089+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:09:09.432089+0000","last_undegraded":"2024-03-16T20:09:09.432089+0000","last_fullsized":"2024-03-16T20:09:09.432089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.482024+0000","last_change":"2024-03-16T19:56:43.466016+0000","last_active":"2024-03-16T20:09:14.482024+0000","last_peered":"2024-03-16T20:09:14.482024+0000","last_clean":"2024-03-16T20:09:14.482024+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:09:14.482024+0000","last_undegraded":"2024-03-16T20:09:14.482024+0000","last_fullsized":"2024-03-16T20:09:14.482024+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:21.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405162+0000","last_change":"2024-03-16T19:56:33.982208+0000","last_active":"2024-03-16T20:09:04.405162+0000","last_peered":"2024-03-16T20:09:04.405162+0000","last_clean":"2024-03-16T20:09:04.405162+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:09:04.405162+0000","last_undegraded":"2024-03-16T20:09:04.405162+0000","last_fullsized":"2024-03-16T20:09:04.405162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:14.173354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432140+0000","last_change":"2024-03-16T19:56:36.309589+0000","last_active":"2024-03-16T20:09:09.432140+0000","last_peered":"2024-03-16T20:09:09.432140+0000","last_clean":"2024-03-16T20:09:09.432140+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:09:09.432140+0000","last_undegraded":"2024-03-16T20:09:09.432140+0000","last_fullsized":"2024-03-16T20:09:09.432140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.481971+0000","last_change":"2024-03-16T19:56:43.466408+0000","last_active":"2024-03-16T20:09:14.481971+0000","last_peered":"2024-03-16T20:09:14.481971+0000","last_clean":"2024-03-16T20:09:14.481971+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:09:14.481971+0000","last_undegraded":"2024-03-16T20:09:14.481971+0000","last_fullsized":"2024-03-16T20:09:14.481971+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:54.530519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448622+0000","last_change":"2024-03-16T19:56:36.305097+0000","last_active":"2024-03-16T20:07:29.448622+0000","last_peered":"2024-03-16T20:07:29.448622+0000","last_clean":"2024-03-16T20:07:29.448622+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:07:29.448622+0000","last_undegraded":"2024-03-16T20:07:29.448622+0000","last_fullsized":"2024-03-16T20:07:29.448622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:13.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.395875+0000","last_change":"2024-03-16T19:56:33.905089+0000","last_active":"2024-03-16T20:07:49.395875+0000","last_peered":"2024-03-16T20:07:49.395875+0000","last_clean":"2024-03-16T20:07:49.395875+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:07:49.395875+0000","last_undegraded":"2024-03-16T20:07:49.395875+0000","last_fullsized":"2024-03-16T20:07:49.395875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:07.820335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473222+0000","last_change":"2024-03-16T19:56:43.437259+0000","last_active":"2024-03-16T20:07:59.473222+0000","last_peered":"2024-03-16T20:07:59.473222+0000","last_clean":"2024-03-16T20:07:59.473222+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:07:59.473222+0000","last_undegraded":"2024-03-16T20:07:59.473222+0000","last_fullsized":"2024-03-16T20:07:59.473222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:08.602038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.880201+0000","last_change":"2024-03-16T20:11:47.880238+0000","last_active":"2024-03-16T20:11:47.880201+0000","last_peered":"2024-03-16T20:11:47.880201+0000","last_clean":"2024-03-16T20:11:47.880201+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:11:47.880201+0000","last_undegraded":"2024-03-16T20:11:47.880201+0000","last_fullsized":"2024-03-16T20:11:47.880201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.880065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:16.609260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.437029+0000","last_change":"2024-03-16T19:56:33.857662+0000","last_active":"2024-03-16T20:11:43.437029+0000","last_peered":"2024-03-16T20:11:43.437029+0000","last_clean":"2024-03-16T20:11:43.437029+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:11:43.437029+0000","last_undegraded":"2024-03-16T20:11:43.437029+0000","last_fullsized":"2024-03-16T20:11:43.437029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473182+0000","last_change":"2024-03-16T19:56:43.466423+0000","last_active":"2024-03-16T20:07:59.473182+0000","last_peered":"2024-03-16T20:07:59.473182+0000","last_clean":"2024-03-16T20:07:59.473182+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:07:59.473182+0000","last_undegraded":"2024-03-16T20:07:59.473182+0000","last_fullsized":"2024-03-16T20:07:59.473182+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:44.237339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.367009+0000","last_change":"2024-03-16T19:56:36.222040+0000","last_active":"2024-03-16T20:11:45.367009+0000","last_peered":"2024-03-16T20:11:45.367009+0000","last_clean":"2024-03-16T20:11:45.367009+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:11:45.367009+0000","last_undegraded":"2024-03-16T20:11:45.367009+0000","last_fullsized":"2024-03-16T20:11:45.367009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.437163+0000","last_change":"2024-03-16T19:56:33.905275+0000","last_active":"2024-03-16T20:11:43.437163+0000","last_peered":"2024-03-16T20:11:43.437163+0000","last_clean":"2024-03-16T20:11:43.437163+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:11:43.437163+0000","last_undegraded":"2024-03-16T20:11:43.437163+0000","last_fullsized":"2024-03-16T20:11:43.437163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:15.655803+0000","last_change":"2024-03-16T19:56:43.496611+0000","last_active":"2024-03-16T20:10:15.655803+0000","last_peered":"2024-03-16T20:10:15.655803+0000","last_clean":"2024-03-16T20:10:15.655803+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:10:15.655803+0000","last_undegraded":"2024-03-16T20:10:15.655803+0000","last_fullsized":"2024-03-16T20:10:15.655803+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.469295+0000","last_change":"2024-03-16T19:56:36.251037+0000","last_active":"2024-03-16T20:07:29.469295+0000","last_peered":"2024-03-16T20:07:29.469295+0000","last_clean":"2024-03-16T20:07:29.469295+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:07:29.469295+0000","last_undegraded":"2024-03-16T20:07:29.469295+0000","last_fullsized":"2024-03-16T20:07:29.469295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:18.088667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:34.408513+0000","last_change":"2024-03-16T19:56:33.967225+0000","last_active":"2024-03-16T20:09:34.408513+0000","last_peered":"2024-03-16T20:09:34.408513+0000","last_clean":"2024-03-16T20:09:34.408513+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:09:34.408513+0000","last_undegraded":"2024-03-16T20:09:34.408513+0000","last_fullsized":"2024-03-16T20:09:34.408513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:59.473238+0000","last_change":"2024-03-16T19:56:43.467747+0000","last_active":"2024-03-16T20:07:59.473238+0000","last_peered":"2024-03-16T20:07:59.473238+0000","last_clean":"2024-03-16T20:07:59.473238+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:07:59.473238+0000","last_undegraded":"2024-03-16T20:07:59.473238+0000","last_fullsized":"2024-03-16T20:07:59.473238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:16.326203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:42.537162+0000","last_change":"2024-03-16T19:56:36.313273+0000","last_active":"2024-03-16T20:11:42.537162+0000","last_peered":"2024-03-16T20:11:42.537162+0000","last_clean":"2024-03-16T20:11:42.537162+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:11:42.537162+0000","last_undegraded":"2024-03-16T20:11:42.537162+0000","last_fullsized":"2024-03-16T20:11:42.537162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":49,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:34.464210+0000","last_change":"2024-03-16T19:56:33.972261+0000","last_active":"2024-03-16T20:09:34.464210+0000","last_peered":"2024-03-16T20:09:34.464210+0000","last_clean":"2024-03-16T20:09:34.464210+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:09:34.464210+0000","last_undegraded":"2024-03-16T20:09:34.464210+0000","last_fullsized":"2024-03-16T20:09:34.464210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.481902+0000","last_change":"2024-03-16T19:56:43.489675+0000","last_active":"2024-03-16T20:09:14.481902+0000","last_peered":"2024-03-16T20:09:14.481902+0000","last_clean":"2024-03-16T20:09:14.481902+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:09:14.481902+0000","last_undegraded":"2024-03-16T20:09:14.481902+0000","last_fullsized":"2024-03-16T20:09:14.481902+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:15.193284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.431142+0000","last_change":"2024-03-16T19:57:41.287408+0000","last_active":"2024-03-16T20:10:43.431142+0000","last_peered":"2024-03-16T20:10:43.431142+0000","last_clean":"2024-03-16T20:10:43.431142+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:10:43.431142+0000","last_undegraded":"2024-03-16T20:10:43.431142+0000","last_fullsized":"2024-03-16T20:10:43.431142+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.437012+0000","last_change":"2024-03-16T19:56:33.975637+0000","last_active":"2024-03-16T20:11:43.437012+0000","last_peered":"2024-03-16T20:11:43.437012+0000","last_clean":"2024-03-16T20:11:43.437012+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:11:43.437012+0000","last_undegraded":"2024-03-16T20:11:43.437012+0000","last_fullsized":"2024-03-16T20:11:43.437012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.488144+0000","last_change":"2024-03-16T19:56:43.492107+0000","last_active":"2024-03-16T20:10:43.488144+0000","last_peered":"2024-03-16T20:10:43.488144+0000","last_clean":"2024-03-16T20:10:43.488144+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:10:43.488144+0000","last_undegraded":"2024-03-16T20:10:43.488144+0000","last_fullsized":"2024-03-16T20:10:43.488144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.948757+0000","last_change":"2024-03-16T20:11:43.948796+0000","last_active":"2024-03-16T20:11:43.948757+0000","last_peered":"2024-03-16T20:11:43.948757+0000","last_clean":"2024-03-16T20:11:43.948757+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:11:43.948757+0000","last_undegraded":"2024-03-16T20:11:43.948757+0000","last_fullsized":"2024-03-16T20:11:43.948757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.948646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:36:25.189231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.405217+0000","last_change":"2024-03-16T19:56:33.967984+0000","last_active":"2024-03-16T20:09:04.405217+0000","last_peered":"2024-03-16T20:09:04.405217+0000","last_clean":"2024-03-16T20:09:04.405217+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:09:04.405217+0000","last_undegraded":"2024-03-16T20:09:04.405217+0000","last_fullsized":"2024-03-16T20:09:04.405217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:27.581866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:18.314359+0000","last_change":"2024-03-16T19:56:43.486882+0000","last_active":"2024-03-16T20:11:18.314359+0000","last_peered":"2024-03-16T20:11:18.314359+0000","last_clean":"2024-03-16T20:11:18.314359+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:11:18.314359+0000","last_undegraded":"2024-03-16T20:11:18.314359+0000","last_fullsized":"2024-03-16T20:11:18.314359+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.432177+0000","last_change":"2024-03-16T19:56:36.307828+0000","last_active":"2024-03-16T20:09:09.432177+0000","last_peered":"2024-03-16T20:09:09.432177+0000","last_clean":"2024-03-16T20:09:09.432177+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:09:09.432177+0000","last_undegraded":"2024-03-16T20:09:09.432177+0000","last_fullsized":"2024-03-16T20:09:09.432177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:10.661990+0000","last_change":"2024-03-16T19:56:33.965000+0000","last_active":"2024-03-16T20:11:10.661990+0000","last_peered":"2024-03-16T20:11:10.661990+0000","last_clean":"2024-03-16T20:11:10.661990+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:11:10.661990+0000","last_undegraded":"2024-03-16T20:11:10.661990+0000","last_fullsized":"2024-03-16T20:11:10.661990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:20.663210+0000","last_change":"2024-03-16T19:56:43.456797+0000","last_active":"2024-03-16T20:11:20.663210+0000","last_peered":"2024-03-16T20:11:20.663210+0000","last_clean":"2024-03-16T20:11:20.663210+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:11:20.663210+0000","last_undegraded":"2024-03-16T20:11:20.663210+0000","last_fullsized":"2024-03-16T20:11:20.663210+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.481151+0000","last_change":"2024-03-16T19:56:36.302874+0000","last_active":"2024-03-16T20:09:09.481151+0000","last_peered":"2024-03-16T20:09:09.481151+0000","last_clean":"2024-03-16T20:09:09.481151+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:09:09.481151+0000","last_undegraded":"2024-03-16T20:09:09.481151+0000","last_fullsized":"2024-03-16T20:09:09.481151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:16.376260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'6","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431117+0000","last_change":"2024-03-16T19:56:33.975901+0000","last_active":"2024-03-16T20:09:04.431117+0000","last_peered":"2024-03-16T20:09:04.431117+0000","last_clean":"2024-03-16T20:09:04.431117+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:09:04.431117+0000","last_undegraded":"2024-03-16T20:09:04.431117+0000","last_fullsized":"2024-03-16T20:09:04.431117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:20.663278+0000","last_change":"2024-03-16T19:56:43.460193+0000","last_active":"2024-03-16T20:11:20.663278+0000","last_peered":"2024-03-16T20:11:20.663278+0000","last_clean":"2024-03-16T20:11:20.663278+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:11:20.663278+0000","last_undegraded":"2024-03-16T20:11:20.663278+0000","last_fullsized":"2024-03-16T20:11:20.663278+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:35.658335+0000","last_change":"2024-03-16T19:56:36.235468+0000","last_active":"2024-03-16T20:10:35.658335+0000","last_peered":"2024-03-16T20:10:35.658335+0000","last_clean":"2024-03-16T20:10:35.658335+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:10:35.658335+0000","last_undegraded":"2024-03-16T20:10:35.658335+0000","last_fullsized":"2024-03-16T20:10:35.658335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"32'8","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.876745+0000","last_change":"2024-03-16T20:11:46.876796+0000","last_active":"2024-03-16T20:11:46.876745+0000","last_peered":"2024-03-16T20:11:46.876745+0000","last_clean":"2024-03-16T20:11:46.876745+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:11:46.876745+0000","last_undegraded":"2024-03-16T20:11:46.876745+0000","last_fullsized":"2024-03-16T20:11:46.876745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.876643+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:35.335725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:45.652576+0000","last_change":"2024-03-16T19:56:43.467157+0000","last_active":"2024-03-16T20:09:45.652576+0000","last_peered":"2024-03-16T20:09:45.652576+0000","last_clean":"2024-03-16T20:09:45.652576+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:09:45.652576+0000","last_undegraded":"2024-03-16T20:09:45.652576+0000","last_fullsized":"2024-03-16T20:09:45.652576+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:13.313780+0000","last_change":"2024-03-16T19:56:36.316800+0000","last_active":"2024-03-16T20:11:13.313780+0000","last_peered":"2024-03-16T20:11:13.313780+0000","last_clean":"2024-03-16T20:11:13.313780+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:11:13.313780+0000","last_undegraded":"2024-03-16T20:11:13.313780+0000","last_fullsized":"2024-03-16T20:11:13.313780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:34.401181+0000","last_change":"2024-03-16T19:56:33.902970+0000","last_active":"2024-03-16T20:08:34.401181+0000","last_peered":"2024-03-16T20:08:34.401181+0000","last_clean":"2024-03-16T20:08:34.401181+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:08:34.401181+0000","last_undegraded":"2024-03-16T20:08:34.401181+0000","last_fullsized":"2024-03-16T20:08:34.401181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:08.631921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:43.412130+0000","last_change":"2024-03-16T19:56:43.495353+0000","last_active":"2024-03-16T20:09:43.412130+0000","last_peered":"2024-03-16T20:09:43.412130+0000","last_clean":"2024-03-16T20:09:43.412130+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:09:43.412130+0000","last_undegraded":"2024-03-16T20:09:43.412130+0000","last_fullsized":"2024-03-16T20:09:43.412130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:13.385270+0000","last_change":"2024-03-16T19:57:41.238831+0000","last_active":"2024-03-16T20:11:13.385270+0000","last_peered":"2024-03-16T20:11:13.385270+0000","last_clean":"2024-03-16T20:11:13.385270+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:11:13.385270+0000","last_undegraded":"2024-03-16T20:11:13.385270+0000","last_fullsized":"2024-03-16T20:11:13.385270+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.431163+0000","last_change":"2024-03-16T19:56:33.975108+0000","last_active":"2024-03-16T20:09:04.431163+0000","last_peered":"2024-03-16T20:09:04.431163+0000","last_clean":"2024-03-16T20:09:04.431163+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:09:04.431163+0000","last_undegraded":"2024-03-16T20:09:04.431163+0000","last_fullsized":"2024-03-16T20:09:04.431163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":197,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396571+0000","last_change":"2024-03-16T19:55:55.092780+0000","last_active":"2024-03-16T20:07:54.396571+0000","last_peered":"2024-03-16T20:07:54.396571+0000","last_clean":"2024-03-16T20:07:54.396571+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:07:54.396571+0000","last_undegraded":"2024-03-16T20:07:54.396571+0000","last_fullsized":"2024-03-16T20:07:54.396571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:55:53.242247+0000","last_clean_scrub_stamp":"2024-03-16T19:55:53.242247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:36.631220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461272+0000","last_change":"2024-03-16T19:56:43.486923+0000","last_active":"2024-03-16T20:09:14.461272+0000","last_peered":"2024-03-16T20:09:14.461272+0000","last_clean":"2024-03-16T20:09:14.461272+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:09:14.461272+0000","last_undegraded":"2024-03-16T20:09:14.461272+0000","last_fullsized":"2024-03-16T20:09:14.461272+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:53.290340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396635+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T20:07:54.396635+0000","last_peered":"2024-03-16T20:07:54.396635+0000","last_clean":"2024-03-16T20:07:54.396635+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:07:54.396635+0000","last_undegraded":"2024-03-16T20:07:54.396635+0000","last_fullsized":"2024-03-16T20:07:54.396635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.907426+0000","last_change":"2024-03-16T20:11:47.907475+0000","last_active":"2024-03-16T20:11:47.907426+0000","last_peered":"2024-03-16T20:11:47.907426+0000","last_clean":"2024-03-16T20:11:47.907426+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:11:47.907426+0000","last_undegraded":"2024-03-16T20:11:47.907426+0000","last_fullsized":"2024-03-16T20:11:47.907426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.907316+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:21:14.912976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.449341+0000","last_change":"2024-03-16T19:56:43.484591+0000","last_active":"2024-03-16T20:07:34.449341+0000","last_peered":"2024-03-16T20:07:34.449341+0000","last_clean":"2024-03-16T20:07:34.449341+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:07:34.449341+0000","last_undegraded":"2024-03-16T20:07:34.449341+0000","last_fullsized":"2024-03-16T20:07:34.449341+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421860+0000","last_change":"2024-03-16T19:56:36.300111+0000","last_active":"2024-03-16T20:07:54.421860+0000","last_peered":"2024-03-16T20:07:54.421860+0000","last_clean":"2024-03-16T20:07:54.421860+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:07:54.421860+0000","last_undegraded":"2024-03-16T20:07:54.421860+0000","last_fullsized":"2024-03-16T20:07:54.421860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"32'4","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.993849+0000","last_change":"2024-03-16T20:11:44.993895+0000","last_active":"2024-03-16T20:11:44.993849+0000","last_peered":"2024-03-16T20:11:44.993849+0000","last_clean":"2024-03-16T20:11:44.993849+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:11:44.993849+0000","last_undegraded":"2024-03-16T20:11:44.993849+0000","last_fullsized":"2024-03-16T20:11:44.993849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4","last_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_deep_scrub":"32'4","last_deep_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.993741+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:31:57.200016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:13.415859+0000","last_change":"2024-03-16T19:56:43.466858+0000","last_active":"2024-03-16T20:10:13.415859+0000","last_peered":"2024-03-16T20:10:13.415859+0000","last_clean":"2024-03-16T20:10:13.415859+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:10:13.415859+0000","last_undegraded":"2024-03-16T20:10:13.415859+0000","last_fullsized":"2024-03-16T20:10:13.415859+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396733+0000","last_change":"2024-03-16T19:56:36.254603+0000","last_active":"2024-03-16T20:07:54.396733+0000","last_peered":"2024-03-16T20:07:54.396733+0000","last_clean":"2024-03-16T20:07:54.396733+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:07:54.396733+0000","last_undegraded":"2024-03-16T20:07:54.396733+0000","last_fullsized":"2024-03-16T20:07:54.396733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:46.183078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"32'7","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.836407+0000","last_change":"2024-03-16T20:11:44.836458+0000","last_active":"2024-03-16T20:11:44.836407+0000","last_peered":"2024-03-16T20:11:44.836407+0000","last_clean":"2024-03-16T20:11:44.836407+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:11:44.836407+0000","last_undegraded":"2024-03-16T20:11:44.836407+0000","last_fullsized":"2024-03-16T20:11:44.836407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.836292+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:01.699700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.419200+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T20:07:34.419200+0000","last_peered":"2024-03-16T20:07:34.419200+0000","last_clean":"2024-03-16T20:07:34.419200+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:07:34.419200+0000","last_undegraded":"2024-03-16T20:07:34.419200+0000","last_fullsized":"2024-03-16T20:07:34.419200+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:26.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396681+0000","last_change":"2024-03-16T19:56:36.251140+0000","last_active":"2024-03-16T20:07:54.396681+0000","last_peered":"2024-03-16T20:07:54.396681+0000","last_clean":"2024-03-16T20:07:54.396681+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:07:54.396681+0000","last_undegraded":"2024-03-16T20:07:54.396681+0000","last_fullsized":"2024-03-16T20:07:54.396681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:51.156788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:49.471915+0000","last_change":"2024-03-16T19:56:33.961302+0000","last_active":"2024-03-16T20:07:49.471915+0000","last_peered":"2024-03-16T20:07:49.471915+0000","last_clean":"2024-03-16T20:07:49.471915+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:07:49.471915+0000","last_undegraded":"2024-03-16T20:07:49.471915+0000","last_fullsized":"2024-03-16T20:07:49.471915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:23.704927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":34,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:34.469980+0000","last_change":"2024-03-16T19:57:41.242958+0000","last_active":"2024-03-16T20:07:34.469980+0000","last_peered":"2024-03-16T20:07:34.469980+0000","last_clean":"2024-03-16T20:07:34.469980+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:07:34.469980+0000","last_undegraded":"2024-03-16T20:07:34.469980+0000","last_fullsized":"2024-03-16T20:07:34.469980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:17:20.536712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399350+0000","last_change":"2024-03-16T19:56:36.256530+0000","last_active":"2024-03-16T20:08:19.399350+0000","last_peered":"2024-03-16T20:08:19.399350+0000","last_clean":"2024-03-16T20:08:19.399350+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:08:19.399350+0000","last_undegraded":"2024-03-16T20:08:19.399350+0000","last_fullsized":"2024-03-16T20:08:19.399350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'26","reported_seq":77,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.873458+0000","last_change":"2024-03-16T20:11:45.873518+0000","last_active":"2024-03-16T20:11:45.873458+0000","last_peered":"2024-03-16T20:11:45.873458+0000","last_clean":"2024-03-16T20:11:45.873458+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:11:45.873458+0000","last_undegraded":"2024-03-16T20:11:45.873458+0000","last_fullsized":"2024-03-16T20:11:45.873458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'26","last_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_deep_scrub":"32'26","last_deep_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.873355+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:46.491674+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:13.313845+0000","last_change":"2024-03-16T19:56:43.466890+0000","last_active":"2024-03-16T20:11:13.313845+0000","last_peered":"2024-03-16T20:11:13.313845+0000","last_clean":"2024-03-16T20:11:13.313845+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:11:13.313845+0000","last_undegraded":"2024-03-16T20:11:13.313845+0000","last_fullsized":"2024-03-16T20:11:13.313845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.455260+0000","last_change":"2024-03-16T19:56:36.316547+0000","last_active":"2024-03-16T20:08:19.455260+0000","last_peered":"2024-03-16T20:08:19.455260+0000","last_clean":"2024-03-16T20:08:19.455260+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:08:19.455260+0000","last_undegraded":"2024-03-16T20:08:19.455260+0000","last_fullsized":"2024-03-16T20:08:19.455260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:49.746181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'6","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454559+0000","last_change":"2024-03-16T19:56:33.975609+0000","last_active":"2024-03-16T20:08:14.454559+0000","last_peered":"2024-03-16T20:08:14.454559+0000","last_clean":"2024-03-16T20:08:14.454559+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:08:14.454559+0000","last_undegraded":"2024-03-16T20:08:14.454559+0000","last_fullsized":"2024-03-16T20:08:14.454559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:27.528537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:13.385318+0000","last_change":"2024-03-16T19:56:43.489922+0000","last_active":"2024-03-16T20:11:13.385318+0000","last_peered":"2024-03-16T20:11:13.385318+0000","last_clean":"2024-03-16T20:11:13.385318+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:11:13.385318+0000","last_undegraded":"2024-03-16T20:11:13.385318+0000","last_fullsized":"2024-03-16T20:11:13.385318+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.399380+0000","last_change":"2024-03-16T19:56:36.250690+0000","last_active":"2024-03-16T20:08:19.399380+0000","last_peered":"2024-03-16T20:08:19.399380+0000","last_clean":"2024-03-16T20:08:19.399380+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:08:19.399380+0000","last_undegraded":"2024-03-16T20:08:19.399380+0000","last_fullsized":"2024-03-16T20:08:19.399380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:40.159143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.424455+0000","last_change":"2024-03-16T19:56:33.970820+0000","last_active":"2024-03-16T20:08:14.424455+0000","last_peered":"2024-03-16T20:08:14.424455+0000","last_clean":"2024-03-16T20:08:14.424455+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:08:14.424455+0000","last_undegraded":"2024-03-16T20:08:14.424455+0000","last_fullsized":"2024-03-16T20:08:14.424455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:43:15.868345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.488235+0000","last_change":"2024-03-16T19:56:43.491677+0000","last_active":"2024-03-16T20:10:43.488235+0000","last_peered":"2024-03-16T20:10:43.488235+0000","last_clean":"2024-03-16T20:10:43.488235+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:10:43.488235+0000","last_undegraded":"2024-03-16T20:10:43.488235+0000","last_fullsized":"2024-03-16T20:10:43.488235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.340430+0000","last_change":"2024-03-16T19:57:41.287279+0000","last_active":"2024-03-16T20:11:43.340430+0000","last_peered":"2024-03-16T20:11:43.340430+0000","last_clean":"2024-03-16T20:11:43.340430+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:11:43.340430+0000","last_undegraded":"2024-03-16T20:11:43.340430+0000","last_fullsized":"2024-03-16T20:11:43.340430+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"32'15","reported_seq":65,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.686996+0000","last_change":"2024-03-16T20:11:43.687053+0000","last_active":"2024-03-16T20:11:43.686996+0000","last_peered":"2024-03-16T20:11:43.686996+0000","last_clean":"2024-03-16T20:11:43.686996+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:11:43.686996+0000","last_undegraded":"2024-03-16T20:11:43.686996+0000","last_fullsized":"2024-03-16T20:11:43.686996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'15","last_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_deep_scrub":"32'15","last_deep_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.686892+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:16.356150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:49.429172+0000","last_change":"2024-03-16T19:56:43.489545+0000","last_active":"2024-03-16T20:08:49.429172+0000","last_peered":"2024-03-16T20:08:49.429172+0000","last_clean":"2024-03-16T20:08:49.429172+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:08:49.429172+0000","last_undegraded":"2024-03-16T20:08:49.429172+0000","last_fullsized":"2024-03-16T20:08:49.429172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:06:56.618499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402419+0000","last_change":"2024-03-16T19:56:36.213455+0000","last_active":"2024-03-16T20:08:44.402419+0000","last_peered":"2024-03-16T20:08:44.402419+0000","last_clean":"2024-03-16T20:08:44.402419+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:08:44.402419+0000","last_undegraded":"2024-03-16T20:08:44.402419+0000","last_fullsized":"2024-03-16T20:08:44.402419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:00.680445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:14.454598+0000","last_change":"2024-03-16T19:56:33.971945+0000","last_active":"2024-03-16T20:08:14.454598+0000","last_peered":"2024-03-16T20:08:14.454598+0000","last_clean":"2024-03-16T20:08:14.454598+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:08:14.454598+0000","last_undegraded":"2024-03-16T20:08:14.454598+0000","last_fullsized":"2024-03-16T20:08:14.454598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:45.659377+0000","last_change":"2024-03-16T19:56:43.439630+0000","last_active":"2024-03-16T20:10:45.659377+0000","last_peered":"2024-03-16T20:10:45.659377+0000","last_clean":"2024-03-16T20:10:45.659377+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:10:45.659377+0000","last_undegraded":"2024-03-16T20:10:45.659377+0000","last_fullsized":"2024-03-16T20:10:45.659377+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.475987+0000","last_change":"2024-03-16T19:56:36.249682+0000","last_active":"2024-03-16T20:08:19.475987+0000","last_peered":"2024-03-16T20:08:19.475987+0000","last_clean":"2024-03-16T20:08:19.475987+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:08:19.475987+0000","last_undegraded":"2024-03-16T20:08:19.475987+0000","last_fullsized":"2024-03-16T20:08:19.475987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:21:01.921057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:05.661442+0000","last_change":"2024-03-16T19:56:33.983480+0000","last_active":"2024-03-16T20:11:05.661442+0000","last_peered":"2024-03-16T20:11:05.661442+0000","last_clean":"2024-03-16T20:11:05.661442+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:11:05.661442+0000","last_undegraded":"2024-03-16T20:11:05.661442+0000","last_fullsized":"2024-03-16T20:11:05.661442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:24.476654+0000","last_change":"2024-03-16T19:56:43.456002+0000","last_active":"2024-03-16T20:08:24.476654+0000","last_peered":"2024-03-16T20:08:24.476654+0000","last_clean":"2024-03-16T20:08:24.476654+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:08:24.476654+0000","last_undegraded":"2024-03-16T20:08:24.476654+0000","last_fullsized":"2024-03-16T20:08:24.476654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:20.502785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.766369+0000","last_change":"2024-03-16T20:11:45.766407+0000","last_active":"2024-03-16T20:11:45.766369+0000","last_peered":"2024-03-16T20:11:45.766369+0000","last_clean":"2024-03-16T20:11:45.766369+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:11:45.766369+0000","last_undegraded":"2024-03-16T20:11:45.766369+0000","last_fullsized":"2024-03-16T20:11:45.766369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.766246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:05.475898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427645+0000","last_change":"2024-03-16T19:56:33.978967+0000","last_active":"2024-03-16T20:08:39.427645+0000","last_peered":"2024-03-16T20:08:39.427645+0000","last_clean":"2024-03-16T20:08:39.427645+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:08:39.427645+0000","last_undegraded":"2024-03-16T20:08:39.427645+0000","last_fullsized":"2024-03-16T20:08:39.427645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.488178+0000","last_change":"2024-03-16T19:57:41.246101+0000","last_active":"2024-03-16T20:10:43.488178+0000","last_peered":"2024-03-16T20:10:43.488178+0000","last_clean":"2024-03-16T20:10:43.488178+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:10:43.488178+0000","last_undegraded":"2024-03-16T20:10:43.488178+0000","last_fullsized":"2024-03-16T20:10:43.488178+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.428389+0000","last_change":"2024-03-16T19:56:36.318424+0000","last_active":"2024-03-16T20:08:44.428389+0000","last_peered":"2024-03-16T20:08:44.428389+0000","last_clean":"2024-03-16T20:08:44.428389+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:08:44.428389+0000","last_undegraded":"2024-03-16T20:08:44.428389+0000","last_fullsized":"2024-03-16T20:08:44.428389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.457446+0000","last_change":"2024-03-16T19:56:33.982685+0000","last_active":"2024-03-16T20:08:39.457446+0000","last_peered":"2024-03-16T20:08:39.457446+0000","last_clean":"2024-03-16T20:08:39.457446+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:08:39.457446+0000","last_undegraded":"2024-03-16T20:08:39.457446+0000","last_fullsized":"2024-03-16T20:08:39.457446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:05.982296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.488204+0000","last_change":"2024-03-16T19:56:43.493528+0000","last_active":"2024-03-16T20:10:43.488204+0000","last_peered":"2024-03-16T20:10:43.488204+0000","last_clean":"2024-03-16T20:10:43.488204+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:10:43.488204+0000","last_undegraded":"2024-03-16T20:10:43.488204+0000","last_fullsized":"2024-03-16T20:10:43.488204+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:43.419120+0000","last_change":"2024-03-16T19:56:43.492259+0000","last_active":"2024-03-16T20:10:43.419120+0000","last_peered":"2024-03-16T20:10:43.419120+0000","last_clean":"2024-03-16T20:10:43.419120+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:10:43.419120+0000","last_undegraded":"2024-03-16T20:10:43.419120+0000","last_fullsized":"2024-03-16T20:10:43.419120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:05.661389+0000","last_change":"2024-03-16T19:56:36.222736+0000","last_active":"2024-03-16T20:11:05.661389+0000","last_peered":"2024-03-16T20:11:05.661389+0000","last_clean":"2024-03-16T20:11:05.661389+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:11:05.661389+0000","last_undegraded":"2024-03-16T20:11:05.661389+0000","last_fullsized":"2024-03-16T20:11:05.661389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:39.427570+0000","last_change":"2024-03-16T19:56:33.977019+0000","last_active":"2024-03-16T20:08:39.427570+0000","last_peered":"2024-03-16T20:08:39.427570+0000","last_clean":"2024-03-16T20:08:39.427570+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:08:39.427570+0000","last_undegraded":"2024-03-16T20:08:39.427570+0000","last_fullsized":"2024-03-16T20:08:39.427570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:40:14.155938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:14.461319+0000","last_change":"2024-03-16T19:57:41.279710+0000","last_active":"2024-03-16T20:09:14.461319+0000","last_peered":"2024-03-16T20:09:14.461319+0000","last_clean":"2024-03-16T20:09:14.461319+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:09:14.461319+0000","last_undegraded":"2024-03-16T20:09:14.461319+0000","last_fullsized":"2024-03-16T20:09:14.461319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:02:54.725387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.883135+0000","last_change":"2024-03-16T20:11:45.883182+0000","last_active":"2024-03-16T20:11:45.883135+0000","last_peered":"2024-03-16T20:11:45.883135+0000","last_clean":"2024-03-16T20:11:45.883135+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:11:45.883135+0000","last_undegraded":"2024-03-16T20:11:45.883135+0000","last_fullsized":"2024-03-16T20:11:45.883135+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.883021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:48.411583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:08.384646+0000","last_change":"2024-03-16T19:56:33.974223+0000","last_active":"2024-03-16T20:11:08.384646+0000","last_peered":"2024-03-16T20:11:08.384646+0000","last_clean":"2024-03-16T20:11:08.384646+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:11:08.384646+0000","last_undegraded":"2024-03-16T20:11:08.384646+0000","last_fullsized":"2024-03-16T20:11:08.384646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:04.459824+0000","last_change":"2024-03-16T19:56:33.975149+0000","last_active":"2024-03-16T20:09:04.459824+0000","last_peered":"2024-03-16T20:09:04.459824+0000","last_clean":"2024-03-16T20:09:04.459824+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:09:04.459824+0000","last_undegraded":"2024-03-16T20:09:04.459824+0000","last_fullsized":"2024-03-16T20:09:04.459824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:59.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.819175+0000","last_change":"2024-03-16T20:11:47.819214+0000","last_active":"2024-03-16T20:11:47.819175+0000","last_peered":"2024-03-16T20:11:47.819175+0000","last_clean":"2024-03-16T20:11:47.819175+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:11:47.819175+0000","last_undegraded":"2024-03-16T20:11:47.819175+0000","last_fullsized":"2024-03-16T20:11:47.819175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.819062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:50.347624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.340463+0000","last_change":"2024-03-16T19:56:43.466134+0000","last_active":"2024-03-16T20:11:43.340463+0000","last_peered":"2024-03-16T20:11:43.340463+0000","last_clean":"2024-03-16T20:11:43.340463+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:11:43.340463+0000","last_undegraded":"2024-03-16T20:11:43.340463+0000","last_fullsized":"2024-03-16T20:11:43.340463+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:33.387307+0000","last_change":"2024-03-16T19:56:33.905190+0000","last_active":"2024-03-16T20:11:33.387307+0000","last_peered":"2024-03-16T20:11:33.387307+0000","last_clean":"2024-03-16T20:11:33.387307+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:11:33.387307+0000","last_undegraded":"2024-03-16T20:11:33.387307+0000","last_fullsized":"2024-03-16T20:11:33.387307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:32.502248+0000","last_clean_scrub_stamp":"2024-03-16T19:56:32.502248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.481114+0000","last_change":"2024-03-16T19:56:36.248322+0000","last_active":"2024-03-16T20:09:09.481114+0000","last_peered":"2024-03-16T20:09:09.481114+0000","last_clean":"2024-03-16T20:09:09.481114+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:09:09.481114+0000","last_undegraded":"2024-03-16T20:09:09.481114+0000","last_fullsized":"2024-03-16T20:09:09.481114+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:14:33.994286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.388426+0000","last_change":"2024-03-16T19:56:43.464463+0000","last_active":"2024-03-16T20:11:43.388426+0000","last_peered":"2024-03-16T20:11:43.388426+0000","last_clean":"2024-03-16T20:11:43.388426+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:11:43.388426+0000","last_undegraded":"2024-03-16T20:11:43.388426+0000","last_fullsized":"2024-03-16T20:11:43.388426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.460475+0000","last_change":"2024-03-16T19:56:36.318687+0000","last_active":"2024-03-16T20:09:09.460475+0000","last_peered":"2024-03-16T20:09:09.460475+0000","last_clean":"2024-03-16T20:09:09.460475+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:09:09.460475+0000","last_undegraded":"2024-03-16T20:09:09.460475+0000","last_fullsized":"2024-03-16T20:09:09.460475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:44:15.541053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:13.337472+0000","last_change":"2024-03-16T19:56:36.299339+0000","last_active":"2024-03-16T20:11:13.337472+0000","last_peered":"2024-03-16T20:11:13.337472+0000","last_clean":"2024-03-16T20:11:13.337472+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:11:13.337472+0000","last_undegraded":"2024-03-16T20:11:13.337472+0000","last_fullsized":"2024-03-16T20:11:13.337472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.431997+0000","last_change":"2024-03-16T19:56:36.306744+0000","last_active":"2024-03-16T20:09:09.431997+0000","last_peered":"2024-03-16T20:09:09.431997+0000","last_clean":"2024-03-16T20:09:09.431997+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:09:09.431997+0000","last_undegraded":"2024-03-16T20:09:09.431997+0000","last_fullsized":"2024-03-16T20:09:09.431997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:25:52.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:18.314396+0000","last_change":"2024-03-16T19:56:36.317863+0000","last_active":"2024-03-16T20:11:18.314396+0000","last_peered":"2024-03-16T20:11:18.314396+0000","last_clean":"2024-03-16T20:11:18.314396+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:11:18.314396+0000","last_undegraded":"2024-03-16T20:11:18.314396+0000","last_fullsized":"2024-03-16T20:11:18.314396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.367170+0000","last_change":"2024-03-16T19:56:36.255600+0000","last_active":"2024-03-16T20:11:45.367170+0000","last_peered":"2024-03-16T20:11:45.367170+0000","last_clean":"2024-03-16T20:11:45.367170+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:11:45.367170+0000","last_undegraded":"2024-03-16T20:11:45.367170+0000","last_fullsized":"2024-03-16T20:11:45.367170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.437490+0000","last_change":"2024-03-16T19:56:36.222925+0000","last_active":"2024-03-16T20:11:43.437490+0000","last_peered":"2024-03-16T20:11:43.437490+0000","last_clean":"2024-03-16T20:11:43.437490+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:11:43.437490+0000","last_undegraded":"2024-03-16T20:11:43.437490+0000","last_fullsized":"2024-03-16T20:11:43.437490+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.448666+0000","last_change":"2024-03-16T19:56:36.318269+0000","last_active":"2024-03-16T20:07:29.448666+0000","last_peered":"2024-03-16T20:07:29.448666+0000","last_clean":"2024-03-16T20:07:29.448666+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:07:29.448666+0000","last_undegraded":"2024-03-16T20:07:29.448666+0000","last_fullsized":"2024-03-16T20:07:29.448666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.367318+0000","last_change":"2024-03-16T19:56:36.257187+0000","last_active":"2024-03-16T20:11:45.367318+0000","last_peered":"2024-03-16T20:11:45.367318+0000","last_clean":"2024-03-16T20:11:45.367318+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:11:45.367318+0000","last_undegraded":"2024-03-16T20:11:45.367318+0000","last_fullsized":"2024-03-16T20:11:45.367318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:29.418412+0000","last_change":"2024-03-16T19:56:36.293492+0000","last_active":"2024-03-16T20:07:29.418412+0000","last_peered":"2024-03-16T20:07:29.418412+0000","last_clean":"2024-03-16T20:07:29.418412+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:07:29.418412+0000","last_undegraded":"2024-03-16T20:07:29.418412+0000","last_fullsized":"2024-03-16T20:07:29.418412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:14.430085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:08.483998+0000","last_change":"2024-03-16T19:56:36.257249+0000","last_active":"2024-03-16T20:10:08.483998+0000","last_peered":"2024-03-16T20:10:08.483998+0000","last_clean":"2024-03-16T20:10:08.483998+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:10:08.483998+0000","last_undegraded":"2024-03-16T20:10:08.483998+0000","last_fullsized":"2024-03-16T20:10:08.483998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.421807+0000","last_change":"2024-03-16T19:56:36.317595+0000","last_active":"2024-03-16T20:07:54.421807+0000","last_peered":"2024-03-16T20:07:54.421807+0000","last_clean":"2024-03-16T20:07:54.421807+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:07:54.421807+0000","last_undegraded":"2024-03-16T20:07:54.421807+0000","last_fullsized":"2024-03-16T20:07:54.421807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:37.388902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:08.415275+0000","last_change":"2024-03-16T19:56:36.324553+0000","last_active":"2024-03-16T20:10:08.415275+0000","last_peered":"2024-03-16T20:10:08.415275+0000","last_clean":"2024-03-16T20:10:08.415275+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:10:08.415275+0000","last_undegraded":"2024-03-16T20:10:08.415275+0000","last_fullsized":"2024-03-16T20:10:08.415275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.452068+0000","last_change":"2024-03-16T19:56:36.305161+0000","last_active":"2024-03-16T20:07:54.452068+0000","last_peered":"2024-03-16T20:07:54.452068+0000","last_clean":"2024-03-16T20:07:54.452068+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:07:54.452068+0000","last_undegraded":"2024-03-16T20:07:54.452068+0000","last_fullsized":"2024-03-16T20:07:54.452068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:46.382384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:07:54.396659+0000","last_change":"2024-03-16T19:56:36.232730+0000","last_active":"2024-03-16T20:07:54.396659+0000","last_peered":"2024-03-16T20:07:54.396659+0000","last_clean":"2024-03-16T20:07:54.396659+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:07:54.396659+0000","last_undegraded":"2024-03-16T20:07:54.396659+0000","last_fullsized":"2024-03-16T20:07:54.396659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:38.430634+0000","last_change":"2024-03-16T19:56:36.310313+0000","last_active":"2024-03-16T20:10:38.430634+0000","last_peered":"2024-03-16T20:10:38.430634+0000","last_clean":"2024-03-16T20:10:38.430634+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:10:38.430634+0000","last_undegraded":"2024-03-16T20:10:38.430634+0000","last_fullsized":"2024-03-16T20:10:38.430634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425243+0000","last_change":"2024-03-16T19:56:36.319394+0000","last_active":"2024-03-16T20:08:19.425243+0000","last_peered":"2024-03-16T20:08:19.425243+0000","last_clean":"2024-03-16T20:08:19.425243+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:08:19.425243+0000","last_undegraded":"2024-03-16T20:08:19.425243+0000","last_fullsized":"2024-03-16T20:08:19.425243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:47.170166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:19.425164+0000","last_change":"2024-03-16T19:56:36.316754+0000","last_active":"2024-03-16T20:08:19.425164+0000","last_peered":"2024-03-16T20:08:19.425164+0000","last_clean":"2024-03-16T20:08:19.425164+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:08:19.425164+0000","last_undegraded":"2024-03-16T20:08:19.425164+0000","last_fullsized":"2024-03-16T20:08:19.425164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:37:47.682082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:35.658366+0000","last_change":"2024-03-16T19:56:36.254460+0000","last_active":"2024-03-16T20:10:35.658366+0000","last_peered":"2024-03-16T20:10:35.658366+0000","last_clean":"2024-03-16T20:10:35.658366+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:10:35.658366+0000","last_undegraded":"2024-03-16T20:10:35.658366+0000","last_fullsized":"2024-03-16T20:10:35.658366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:38.418551+0000","last_change":"2024-03-16T19:56:36.316623+0000","last_active":"2024-03-16T20:10:38.418551+0000","last_peered":"2024-03-16T20:10:38.418551+0000","last_clean":"2024-03-16T20:10:38.418551+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:10:38.418551+0000","last_undegraded":"2024-03-16T20:10:38.418551+0000","last_fullsized":"2024-03-16T20:10:38.418551+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:08.384673+0000","last_change":"2024-03-16T19:56:36.306933+0000","last_active":"2024-03-16T20:11:08.384673+0000","last_peered":"2024-03-16T20:11:08.384673+0000","last_clean":"2024-03-16T20:11:08.384673+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:11:08.384673+0000","last_undegraded":"2024-03-16T20:11:08.384673+0000","last_fullsized":"2024-03-16T20:11:08.384673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.478645+0000","last_change":"2024-03-16T19:56:36.305247+0000","last_active":"2024-03-16T20:08:44.478645+0000","last_peered":"2024-03-16T20:08:44.478645+0000","last_clean":"2024-03-16T20:08:44.478645+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:08:44.478645+0000","last_undegraded":"2024-03-16T20:08:44.478645+0000","last_fullsized":"2024-03-16T20:08:44.478645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:29.748860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457937+0000","last_change":"2024-03-16T19:56:36.298523+0000","last_active":"2024-03-16T20:08:44.457937+0000","last_peered":"2024-03-16T20:08:44.457937+0000","last_clean":"2024-03-16T20:08:44.457937+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:08:44.457937+0000","last_undegraded":"2024-03-16T20:08:44.457937+0000","last_fullsized":"2024-03-16T20:08:44.457937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:53:08.344847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.457968+0000","last_change":"2024-03-16T19:56:36.307920+0000","last_active":"2024-03-16T20:08:44.457968+0000","last_peered":"2024-03-16T20:08:44.457968+0000","last_clean":"2024-03-16T20:08:44.457968+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:08:44.457968+0000","last_undegraded":"2024-03-16T20:08:44.457968+0000","last_fullsized":"2024-03-16T20:08:44.457968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:21.283082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:08:44.402472+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T20:08:44.402472+0000","last_peered":"2024-03-16T20:08:44.402472+0000","last_clean":"2024-03-16T20:08:44.402472+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:08:44.402472+0000","last_undegraded":"2024-03-16T20:08:44.402472+0000","last_fullsized":"2024-03-16T20:08:44.402472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:23.525240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:38.387830+0000","last_change":"2024-03-16T19:56:36.307772+0000","last_active":"2024-03-16T20:11:38.387830+0000","last_peered":"2024-03-16T20:11:38.387830+0000","last_clean":"2024-03-16T20:11:38.387830+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:11:38.387830+0000","last_undegraded":"2024-03-16T20:11:38.387830+0000","last_fullsized":"2024-03-16T20:11:38.387830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405729+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T20:09:09.405729+0000","last_peered":"2024-03-16T20:09:09.405729+0000","last_clean":"2024-03-16T20:09:09.405729+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:09:09.405729+0000","last_undegraded":"2024-03-16T20:09:09.405729+0000","last_fullsized":"2024-03-16T20:09:09.405729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:01.753601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.431944+0000","last_change":"2024-03-16T19:56:36.318639+0000","last_active":"2024-03-16T20:09:09.431944+0000","last_peered":"2024-03-16T20:09:09.431944+0000","last_clean":"2024-03-16T20:09:09.431944+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:09:09.431944+0000","last_undegraded":"2024-03-16T20:09:09.431944+0000","last_fullsized":"2024-03-16T20:09:09.431944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:50:09.477766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405762+0000","last_change":"2024-03-16T19:56:36.232264+0000","last_active":"2024-03-16T20:09:09.405762+0000","last_peered":"2024-03-16T20:09:09.405762+0000","last_clean":"2024-03-16T20:09:09.405762+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:09:09.405762+0000","last_undegraded":"2024-03-16T20:09:09.405762+0000","last_fullsized":"2024-03-16T20:09:09.405762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:46.887520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.393199+0000","last_change":"2024-03-16T19:57:41.236414+0000","last_active":"2024-03-16T20:11:44.393199+0000","last_peered":"2024-03-16T20:11:44.393199+0000","last_clean":"2024-03-16T20:11:44.393199+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:11:44.393199+0000","last_undegraded":"2024-03-16T20:11:44.393199+0000","last_fullsized":"2024-03-16T20:11:44.393199+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.718717+0000","last_change":"2024-03-16T20:11:44.718754+0000","last_active":"2024-03-16T20:11:44.718717+0000","last_peered":"2024-03-16T20:11:44.718717+0000","last_clean":"2024-03-16T20:11:44.718717+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:11:44.718717+0000","last_undegraded":"2024-03-16T20:11:44.718717+0000","last_fullsized":"2024-03-16T20:11:44.718717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.718600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:50.182736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:10.662019+0000","last_change":"2024-03-16T19:56:36.316339+0000","last_active":"2024-03-16T20:11:10.662019+0000","last_peered":"2024-03-16T20:11:10.662019+0000","last_clean":"2024-03-16T20:11:10.662019+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:11:10.662019+0000","last_undegraded":"2024-03-16T20:11:10.662019+0000","last_fullsized":"2024-03-16T20:11:10.662019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:09:09.405802+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T20:09:09.405802+0000","last_peered":"2024-03-16T20:09:09.405802+0000","last_clean":"2024-03-16T20:09:09.405802+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:09:09.405802+0000","last_undegraded":"2024-03-16T20:09:09.405802+0000","last_fullsized":"2024-03-16T20:09:09.405802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:46.441903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476931,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476933,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476932,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476930,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509634,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509636,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509636,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542339,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.129 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.130 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.131 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.132 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-16T19:55:53.242247+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=55, tm_sec=53, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.133 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.134 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.135 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.136 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-16T19:56:32.502248+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=32, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.137 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.138 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.139 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:11:50.140 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T20:12:02.760 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:46.838 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.853 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:46.931 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.943 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:46.958 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.949 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:46.964 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.965 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:46.981 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.979 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:16:47.057 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:02.981 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:46.997 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:03.004 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:16:47.020 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:03.034 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:16:47.049 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:12:10.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:12:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 Processor -- start 2024-03-16T20:12:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 -- start start 2024-03-16T20:12:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 0x7f8518151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85180c7830 0x7f8518152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8518157eb0 con 0x7f85180cdb80 2024-03-16T20:12:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85180c4c80 con 0x7f85180c7830 2024-03-16T20:12:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851ec41640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851806fa80 con 0x7f85180cdf60 2024-03-16T20:12:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f8517fff640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 0x7f8518151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851c9dd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f85180cdb80 0x7f851806f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54240/0 (socket says 172.21.15.32:54240) 2024-03-16T20:12:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851c9dd640 1 -- 172.21.15.32:0/3870160061 learned_addr learned my addr 172.21.15.32:0/3870160061 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:12:10.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.255+0000 7f851d1de640 1 --2- 172.21.15.32:0/3870160061 >> v2:172.21.15.59:3300/0 conn(0x7f85180c7830 0x7f8518152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f85177fe640 1 -- 172.21.15.32:0/3870160061 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286748572 0 0) 0x7f8518157eb0 con 0x7f85180cdb80 2024-03-16T20:12:10.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f85177fe640 1 -- 172.21.15.32:0/3870160061 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8508003530 con 0x7f85180cdb80 2024-03-16T20:12:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f8517fff640 1 -- 172.21.15.32:0/3870160061 >> v2:172.21.15.59:3300/0 conn(0x7f85180c7830 msgr2=0x7f8518152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f8517fff640 1 --2- 172.21.15.32:0/3870160061 >> v2:172.21.15.59:3300/0 conn(0x7f85180c7830 0x7f8518152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f8517fff640 1 -- 172.21.15.32:0/3870160061 >> v1:172.21.15.32:6789/0 conn(0x7f85180cdb80 legacy=0x7f851806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.259+0000 7f8517fff640 1 -- 172.21.15.32:0/3870160061 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180c4f40 con 0x7f85180cdf60 2024-03-16T20:12:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.263+0000 7f8517fff640 1 --2- 172.21.15.32:0/3870160061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 0x7f8518151b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85040097c0 tx=0x7f850401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3070260d64e2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.271+0000 7f85177fe640 1 -- 172.21.15.32:0/3870160061 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040ab670 con 0x7f85180cdf60 2024-03-16T20:12:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.271+0000 7f85177fe640 1 -- 172.21.15.32:0/3870160061 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85040ab7d0 con 0x7f85180cdf60 2024-03-16T20:12:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.271+0000 7f85177fe640 1 -- 172.21.15.32:0/3870160061 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040abb40 con 0x7f85180cdf60 2024-03-16T20:12:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.271+0000 7f851ec41640 1 -- 172.21.15.32:0/3870160061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 msgr2=0x7f8518151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.271+0000 7f851ec41640 1 --2- 172.21.15.32:0/3870160061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 0x7f8518151b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85040097c0 tx=0x7f850401c470 comp rx=0 tx=0).stop 2024-03-16T20:12:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- 172.21.15.32:0/3870160061 shutdown_connections 2024-03-16T20:12:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 --2- 172.21.15.32:0/3870160061 >> v2:172.21.15.59:3300/0 conn(0x7f85180c7830 0x7f8518152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 --2- 172.21.15.32:0/3870160061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdf60 0x7f8518151b20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- 172.21.15.32:0/3870160061 >> 172.21.15.32:0/3870160061 conn(0x7f8518066ea0 msgr2=0x7f85180672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- 172.21.15.32:0/3870160061 shutdown_connections 2024-03-16T20:12:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- 172.21.15.32:0/3870160061 wait complete. 2024-03-16T20:12:10.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 Processor -- start 2024-03-16T20:12:10.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- start start 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85180c7830 0x7f85181f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdb80 0x7f85180cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85180cc990 con 0x7f85180cdf60 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85180ccc40 con 0x7f85180c7830 2024-03-16T20:12:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851ec41640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8518154750 con 0x7f85180cdb80 2024-03-16T20:12:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.275+0000 7f851d1de640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.279+0000 7f851d1de640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39464/0 (socket says 172.21.15.32:39464) 2024-03-16T20:12:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.279+0000 7f851d1de640 1 -- 172.21.15.32:0/1607540327 learned_addr learned my addr 172.21.15.32:0/1607540327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:12:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.279+0000 7f8517fff640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdb80 0x7f85180cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.279+0000 7f851c9dd640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85180c7830 0x7f85181f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.283+0000 7f851d1de640 1 -- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdb80 msgr2=0x7f85180cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.283+0000 7f851d1de640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdb80 0x7f85180cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.283+0000 7f851d1de640 1 -- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85180c7830 msgr2=0x7f85181f0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.283+0000 7f851d1de640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85180c7830 0x7f85181f0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.283+0000 7f851d1de640 1 -- 172.21.15.32:0/1607540327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504009420 con 0x7f85180cdf60 2024-03-16T20:12:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.287+0000 7f851d1de640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f851000efc0 tx=0x7f851000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510009ba0 con 0x7f85180cdf60 2024-03-16T20:12:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851000f040 con 0x7f85180cdf60 2024-03-16T20:12:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85100146c0 con 0x7f85180cdf60 2024-03-16T20:12:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8518154a30 con 0x7f85180cdf60 2024-03-16T20:12:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8518154ff0 con 0x7f85180cdf60 2024-03-16T20:12:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.291+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8518068000 con 0x7f85180cdf60 2024-03-16T20:12:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.595+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8510014820 con 0x7f85180cdf60 2024-03-16T20:12:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.595+0000 7f85157fa640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 0x7f84fc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.595+0000 7f851c9dd640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 0x7f84fc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:10.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.595+0000 7f851c9dd640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 0x7f84fc069350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f850c007fd0 tx=0x7f850c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:10.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.619+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f85100567d0 con 0x7f85180cdf60 2024-03-16T20:12:10.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.619+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f8510052a70 con 0x7f85180cdf60 2024-03-16T20:12:10.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.827+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f851806ea00 con 0x7f84fc066e90 2024-03-16T20:12:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.847+0000 7f85157fa640 1 -- 172.21.15.32:0/1607540327 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332605 (secure 0 0 0) 0x7f851806ea00 con 0x7f84fc066e90 2024-03-16T20:12:10.843 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:12:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:12:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 msgr2=0x7f84fc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 0x7f84fc069350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f850c007fd0 tx=0x7f850c009040 comp rx=0 tx=0).stop 2024-03-16T20:12:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 msgr2=0x7f8518145040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f851000efc0 tx=0x7f851000c490 comp rx=0 tx=0).stop 2024-03-16T20:12:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 shutdown_connections 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f84fc066e90 0x7f84fc069350 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180cdf60 0x7f8518145040 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85180cdb80 0x7f85180cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 --2- 172.21.15.32:0/1607540327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85180c7830 0x7f85181f0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 >> 172.21.15.32:0/1607540327 conn(0x7f8518066ea0 msgr2=0x7f8518067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 shutdown_connections 2024-03-16T20:12:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:10.851+0000 7f851ec41640 1 -- 172.21.15.32:0/1607540327 wait complete. 2024-03-16T20:12:10.862 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-03-16T20:12:08.854839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241000,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733720,"statfs":{"total":773094113280,"available":772847329280,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003600"},"pg_stats":[{"pgid":"2.1f","version":"32'20","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:42.724708+0000","last_change":"2024-03-16T20:11:42.724767+0000","last_active":"2024-03-16T20:11:42.724708+0000","last_peered":"2024-03-16T20:11:42.724708+0000","last_clean":"2024-03-16T20:11:42.724708+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:11:42.724708+0000","last_undegraded":"2024-03-16T20:11:42.724708+0000","last_fullsized":"2024-03-16T20:11:42.724708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_clean_scrub_stamp":"2024-03-16T20:11:42.724584+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:20.296590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.927855+0000","last_change":"2024-03-16T20:11:46.927899+0000","last_active":"2024-03-16T20:11:46.927855+0000","last_peered":"2024-03-16T20:11:46.927855+0000","last_clean":"2024-03-16T20:11:46.927855+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:11:46.927855+0000","last_undegraded":"2024-03-16T20:11:46.927855+0000","last_fullsized":"2024-03-16T20:11:46.927855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.927737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.399583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.913457+0000","last_change":"2024-03-16T20:12:02.913500+0000","last_active":"2024-03-16T20:12:02.913457+0000","last_peered":"2024-03-16T20:12:02.913457+0000","last_clean":"2024-03-16T20:12:02.913457+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:12:02.913457+0000","last_undegraded":"2024-03-16T20:12:02.913457+0000","last_fullsized":"2024-03-16T20:12:02.913457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.913346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.913346+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.913346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:24.865908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.850227+0000","last_change":"2024-03-16T20:11:50.850272+0000","last_active":"2024-03-16T20:11:50.850227+0000","last_peered":"2024-03-16T20:11:50.850227+0000","last_clean":"2024-03-16T20:11:50.850227+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:11:50.850227+0000","last_undegraded":"2024-03-16T20:11:50.850227+0000","last_fullsized":"2024-03-16T20:11:50.850227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3","last_scrub_stamp":"2024-03-16T20:11:50.850119+0000","last_deep_scrub":"32'3","last_deep_scrub_stamp":"2024-03-16T20:11:50.850119+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.850119+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:06.272846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.783367+0000","last_change":"2024-03-16T20:11:46.783404+0000","last_active":"2024-03-16T20:11:46.783367+0000","last_peered":"2024-03-16T20:11:46.783367+0000","last_clean":"2024-03-16T20:11:46.783367+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:11:46.783367+0000","last_undegraded":"2024-03-16T20:11:46.783367+0000","last_fullsized":"2024-03-16T20:11:46.783367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.783256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:53.479114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.996122+0000","last_change":"2024-03-16T20:11:54.996183+0000","last_active":"2024-03-16T20:11:54.996122+0000","last_peered":"2024-03-16T20:11:54.996122+0000","last_clean":"2024-03-16T20:11:54.996122+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:11:54.996122+0000","last_undegraded":"2024-03-16T20:11:54.996122+0000","last_fullsized":"2024-03-16T20:11:54.996122+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.995939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.995939+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.995939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:57.524502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'20","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.361012+0000","last_change":"2024-03-16T20:11:57.361084+0000","last_active":"2024-03-16T20:11:57.361012+0000","last_peered":"2024-03-16T20:11:57.361012+0000","last_clean":"2024-03-16T20:11:57.361012+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:11:57.361012+0000","last_undegraded":"2024-03-16T20:11:57.361012+0000","last_fullsized":"2024-03-16T20:11:57.361012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:57.360851+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:57.360851+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.360851+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:27.227797+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.189988+0000","last_change":"2024-03-16T20:11:57.190023+0000","last_active":"2024-03-16T20:11:57.189988+0000","last_peered":"2024-03-16T20:11:57.189988+0000","last_clean":"2024-03-16T20:11:57.189988+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:11:57.189988+0000","last_undegraded":"2024-03-16T20:11:57.189988+0000","last_fullsized":"2024-03-16T20:11:57.189988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.189894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.189894+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.189894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:40:18.144661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.959804+0000","last_change":"2024-03-16T20:11:55.959847+0000","last_active":"2024-03-16T20:11:55.959804+0000","last_peered":"2024-03-16T20:11:55.959804+0000","last_clean":"2024-03-16T20:11:55.959804+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:11:55.959804+0000","last_undegraded":"2024-03-16T20:11:55.959804+0000","last_fullsized":"2024-03-16T20:11:55.959804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.959692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.959692+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.959692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:19.431538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.400540+0000","last_change":"2024-03-16T20:11:58.400593+0000","last_active":"2024-03-16T20:11:58.400540+0000","last_peered":"2024-03-16T20:11:58.400540+0000","last_clean":"2024-03-16T20:11:58.400540+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:11:58.400540+0000","last_undegraded":"2024-03-16T20:11:58.400540+0000","last_fullsized":"2024-03-16T20:11:58.400540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:58.400442+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:58.400442+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.400442+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:13.823169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.191318+0000","last_change":"2024-03-16T20:11:58.191355+0000","last_active":"2024-03-16T20:11:58.191318+0000","last_peered":"2024-03-16T20:11:58.191318+0000","last_clean":"2024-03-16T20:11:58.191318+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:11:58.191318+0000","last_undegraded":"2024-03-16T20:11:58.191318+0000","last_fullsized":"2024-03-16T20:11:58.191318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.191205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.191205+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.191205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:14:52.421910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.007217+0000","last_change":"2024-03-16T20:11:57.007260+0000","last_active":"2024-03-16T20:11:57.007217+0000","last_peered":"2024-03-16T20:11:57.007217+0000","last_clean":"2024-03-16T20:11:57.007217+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:11:57.007217+0000","last_undegraded":"2024-03-16T20:11:57.007217+0000","last_fullsized":"2024-03-16T20:11:57.007217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.007105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.007105+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.007105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:57.874060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.179505+0000","last_change":"2024-03-16T20:11:55.179632+0000","last_active":"2024-03-16T20:11:55.179505+0000","last_peered":"2024-03-16T20:11:55.179505+0000","last_clean":"2024-03-16T20:11:55.179505+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:11:55.179505+0000","last_undegraded":"2024-03-16T20:11:55.179505+0000","last_fullsized":"2024-03-16T20:11:55.179505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.179204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.179204+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.179204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:03:33.091999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"32'6","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.313926+0000","last_change":"2024-03-16T20:11:54.314000+0000","last_active":"2024-03-16T20:11:54.313926+0000","last_peered":"2024-03-16T20:11:54.313926+0000","last_clean":"2024-03-16T20:11:54.313926+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:11:54.313926+0000","last_undegraded":"2024-03-16T20:11:54.313926+0000","last_fullsized":"2024-03-16T20:11:54.313926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-03-16T20:11:54.313753+0000","last_deep_scrub":"32'6","last_deep_scrub_stamp":"2024-03-16T20:11:54.313753+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.313753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:17.785599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.941678+0000","last_change":"2024-03-16T20:11:52.941717+0000","last_active":"2024-03-16T20:11:52.941678+0000","last_peered":"2024-03-16T20:11:52.941678+0000","last_clean":"2024-03-16T20:11:52.941678+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:11:52.941678+0000","last_undegraded":"2024-03-16T20:11:52.941678+0000","last_fullsized":"2024-03-16T20:11:52.941678+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.941573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.941573+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.941573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:02:20.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.880201+0000","last_change":"2024-03-16T20:11:47.880238+0000","last_active":"2024-03-16T20:11:47.880201+0000","last_peered":"2024-03-16T20:11:47.880201+0000","last_clean":"2024-03-16T20:11:47.880201+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:11:47.880201+0000","last_undegraded":"2024-03-16T20:11:47.880201+0000","last_fullsized":"2024-03-16T20:11:47.880201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.880065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:16.609260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"32'5","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.020055+0000","last_change":"2024-03-16T20:11:46.020079+0000","last_active":"2024-03-16T20:11:46.020055+0000","last_peered":"2024-03-16T20:11:46.020055+0000","last_clean":"2024-03-16T20:11:46.020055+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:11:46.020055+0000","last_undegraded":"2024-03-16T20:11:46.020055+0000","last_fullsized":"2024-03-16T20:11:46.020055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:46.019995+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:46.019995+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.019995+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:08:53.109964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.988661+0000","last_change":"2024-03-16T20:11:57.988704+0000","last_active":"2024-03-16T20:11:57.988661+0000","last_peered":"2024-03-16T20:11:57.988661+0000","last_clean":"2024-03-16T20:11:57.988661+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:11:57.988661+0000","last_undegraded":"2024-03-16T20:11:57.988661+0000","last_fullsized":"2024-03-16T20:11:57.988661+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.988554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.988554+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.988554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:15.925804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.918364+0000","last_change":"2024-03-16T20:11:52.918397+0000","last_active":"2024-03-16T20:11:52.918364+0000","last_peered":"2024-03-16T20:11:52.918364+0000","last_clean":"2024-03-16T20:11:52.918364+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:11:52.918364+0000","last_undegraded":"2024-03-16T20:11:52.918364+0000","last_fullsized":"2024-03-16T20:11:52.918364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.918271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.918271+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.918271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:01.131831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.066808+0000","last_change":"2024-03-16T20:11:47.066963+0000","last_active":"2024-03-16T20:11:47.066808+0000","last_peered":"2024-03-16T20:11:47.066808+0000","last_clean":"2024-03-16T20:11:47.066808+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:11:47.066808+0000","last_undegraded":"2024-03-16T20:11:47.066808+0000","last_fullsized":"2024-03-16T20:11:47.066808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:47.066603+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:47.066603+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.066603+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:51:21.887232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.914995+0000","last_change":"2024-03-16T20:11:59.915147+0000","last_active":"2024-03-16T20:11:59.914995+0000","last_peered":"2024-03-16T20:11:59.914995+0000","last_clean":"2024-03-16T20:11:59.914995+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:11:59.914995+0000","last_undegraded":"2024-03-16T20:11:59.914995+0000","last_fullsized":"2024-03-16T20:11:59.914995+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.914883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.914883+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.914883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:12.338529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.003474+0000","last_change":"2024-03-16T20:11:50.003514+0000","last_active":"2024-03-16T20:11:50.003474+0000","last_peered":"2024-03-16T20:11:50.003474+0000","last_clean":"2024-03-16T20:11:50.003474+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:11:50.003474+0000","last_undegraded":"2024-03-16T20:11:50.003474+0000","last_fullsized":"2024-03-16T20:11:50.003474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.003363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.003363+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.003363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:12:20.749265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"32'3","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.357139+0000","last_change":"2024-03-16T20:11:55.357185+0000","last_active":"2024-03-16T20:11:55.357139+0000","last_peered":"2024-03-16T20:11:55.357139+0000","last_clean":"2024-03-16T20:11:55.357139+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:11:55.357139+0000","last_undegraded":"2024-03-16T20:11:55.357139+0000","last_fullsized":"2024-03-16T20:11:55.357139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3","last_scrub_stamp":"2024-03-16T20:11:55.357043+0000","last_deep_scrub":"32'3","last_deep_scrub_stamp":"2024-03-16T20:11:55.357043+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.357043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:56.223998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.961749+0000","last_change":"2024-03-16T20:11:58.961790+0000","last_active":"2024-03-16T20:11:58.961749+0000","last_peered":"2024-03-16T20:11:58.961749+0000","last_clean":"2024-03-16T20:11:58.961749+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:11:58.961749+0000","last_undegraded":"2024-03-16T20:11:58.961749+0000","last_fullsized":"2024-03-16T20:11:58.961749+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.961630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.961630+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.961630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:28.828577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.797979+0000","last_change":"2024-03-16T20:11:48.798017+0000","last_active":"2024-03-16T20:11:48.797979+0000","last_peered":"2024-03-16T20:11:48.797979+0000","last_clean":"2024-03-16T20:11:48.797979+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:11:48.797979+0000","last_undegraded":"2024-03-16T20:11:48.797979+0000","last_fullsized":"2024-03-16T20:11:48.797979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.797866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.797866+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.797866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:12.269712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.185301+0000","last_change":"2024-03-16T20:11:50.185372+0000","last_active":"2024-03-16T20:11:50.185301+0000","last_peered":"2024-03-16T20:11:50.185301+0000","last_clean":"2024-03-16T20:11:50.185301+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:11:50.185301+0000","last_undegraded":"2024-03-16T20:11:50.185301+0000","last_fullsized":"2024-03-16T20:11:50.185301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:50.185142+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:50.185142+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.185142+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:27.757147+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.955818+0000","last_change":"2024-03-16T20:11:59.955860+0000","last_active":"2024-03-16T20:11:59.955818+0000","last_peered":"2024-03-16T20:11:59.955818+0000","last_clean":"2024-03-16T20:11:59.955818+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:11:59.955818+0000","last_undegraded":"2024-03-16T20:11:59.955818+0000","last_fullsized":"2024-03-16T20:11:59.955818+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.955695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.955695+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.955695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:15.378423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.800890+0000","last_change":"2024-03-16T20:11:53.800929+0000","last_active":"2024-03-16T20:11:53.800890+0000","last_peered":"2024-03-16T20:11:53.800890+0000","last_clean":"2024-03-16T20:11:53.800890+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:11:53.800890+0000","last_undegraded":"2024-03-16T20:11:53.800890+0000","last_fullsized":"2024-03-16T20:11:53.800890+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.800778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.800778+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.800778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:00.496635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.134129+0000","last_change":"2024-03-16T20:11:49.134180+0000","last_active":"2024-03-16T20:11:49.134129+0000","last_peered":"2024-03-16T20:11:49.134129+0000","last_clean":"2024-03-16T20:11:49.134129+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:11:49.134129+0000","last_undegraded":"2024-03-16T20:11:49.134129+0000","last_fullsized":"2024-03-16T20:11:49.134129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:49.134032+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:49.134032+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.134032+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:05.343809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.825013+0000","last_change":"2024-03-16T20:12:00.825054+0000","last_active":"2024-03-16T20:12:00.825013+0000","last_peered":"2024-03-16T20:12:00.825013+0000","last_clean":"2024-03-16T20:12:00.825013+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:12:00.825013+0000","last_undegraded":"2024-03-16T20:12:00.825013+0000","last_fullsized":"2024-03-16T20:12:00.825013+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.824902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.824902+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.824902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:37.127249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.948757+0000","last_change":"2024-03-16T20:11:43.948796+0000","last_active":"2024-03-16T20:11:43.948757+0000","last_peered":"2024-03-16T20:11:43.948757+0000","last_clean":"2024-03-16T20:11:43.948757+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:11:43.948757+0000","last_undegraded":"2024-03-16T20:11:43.948757+0000","last_fullsized":"2024-03-16T20:11:43.948757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.948646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:36:25.189231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"32'7","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.344682+0000","last_change":"2024-03-16T20:11:56.344733+0000","last_active":"2024-03-16T20:11:56.344682+0000","last_peered":"2024-03-16T20:11:56.344682+0000","last_clean":"2024-03-16T20:11:56.344682+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:11:56.344682+0000","last_undegraded":"2024-03-16T20:11:56.344682+0000","last_fullsized":"2024-03-16T20:11:56.344682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:56.344588+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:56.344588+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.344588+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:14.281837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.828724+0000","last_change":"2024-03-16T20:11:55.828765+0000","last_active":"2024-03-16T20:11:55.828724+0000","last_peered":"2024-03-16T20:11:55.828724+0000","last_clean":"2024-03-16T20:11:55.828724+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:11:55.828724+0000","last_undegraded":"2024-03-16T20:11:55.828724+0000","last_fullsized":"2024-03-16T20:11:55.828724+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.828618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.828618+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.828618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:42.946999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.285214+0000","last_change":"2024-03-16T20:11:48.285251+0000","last_active":"2024-03-16T20:11:48.285214+0000","last_peered":"2024-03-16T20:11:48.285214+0000","last_clean":"2024-03-16T20:11:48.285214+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:11:48.285214+0000","last_undegraded":"2024-03-16T20:11:48.285214+0000","last_fullsized":"2024-03-16T20:11:48.285214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.285097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.285097+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.285097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:09.534808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":52,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.089469+0000","last_change":"2024-03-16T20:11:48.089513+0000","last_active":"2024-03-16T20:11:48.089469+0000","last_peered":"2024-03-16T20:11:48.089469+0000","last_clean":"2024-03-16T20:11:48.089469+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:11:48.089469+0000","last_undegraded":"2024-03-16T20:11:48.089469+0000","last_fullsized":"2024-03-16T20:11:48.089469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'1","last_scrub_stamp":"2024-03-16T20:11:48.089355+0000","last_deep_scrub":"29'1","last_deep_scrub_stamp":"2024-03-16T20:11:48.089355+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.089355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:22:35.616024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.977722+0000","last_change":"2024-03-16T20:11:56.977766+0000","last_active":"2024-03-16T20:11:56.977722+0000","last_peered":"2024-03-16T20:11:56.977722+0000","last_clean":"2024-03-16T20:11:56.977722+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:11:56.977722+0000","last_undegraded":"2024-03-16T20:11:56.977722+0000","last_fullsized":"2024-03-16T20:11:56.977722+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.977595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.977595+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.977595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:58:33.889972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.985107+0000","last_change":"2024-03-16T20:11:50.985145+0000","last_active":"2024-03-16T20:11:50.985107+0000","last_peered":"2024-03-16T20:11:50.985107+0000","last_clean":"2024-03-16T20:11:50.985107+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:11:50.985107+0000","last_undegraded":"2024-03-16T20:11:50.985107+0000","last_fullsized":"2024-03-16T20:11:50.985107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.985004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.985004+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.985004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:31.441667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.207803+0000","last_change":"2024-03-16T20:11:52.207870+0000","last_active":"2024-03-16T20:11:52.207803+0000","last_peered":"2024-03-16T20:11:52.207803+0000","last_clean":"2024-03-16T20:11:52.207803+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:11:52.207803+0000","last_undegraded":"2024-03-16T20:11:52.207803+0000","last_fullsized":"2024-03-16T20:11:52.207803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:52.207634+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:52.207634+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.207634+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:51:47.529032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.963647+0000","last_change":"2024-03-16T20:11:57.963689+0000","last_active":"2024-03-16T20:11:57.963647+0000","last_peered":"2024-03-16T20:11:57.963647+0000","last_clean":"2024-03-16T20:11:57.963647+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:11:57.963647+0000","last_undegraded":"2024-03-16T20:11:57.963647+0000","last_fullsized":"2024-03-16T20:11:57.963647+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.963536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.963536+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.963536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:03:09.515499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.011073+0000","last_change":"2024-03-16T20:11:53.011109+0000","last_active":"2024-03-16T20:11:53.011073+0000","last_peered":"2024-03-16T20:11:53.011073+0000","last_clean":"2024-03-16T20:11:53.011073+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:11:53.011073+0000","last_undegraded":"2024-03-16T20:11:53.011073+0000","last_fullsized":"2024-03-16T20:11:53.011073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.010964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.010964+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.010964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:19.139131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"32'8","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.876745+0000","last_change":"2024-03-16T20:11:46.876796+0000","last_active":"2024-03-16T20:11:46.876745+0000","last_peered":"2024-03-16T20:11:46.876745+0000","last_clean":"2024-03-16T20:11:46.876745+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:11:46.876745+0000","last_undegraded":"2024-03-16T20:11:46.876745+0000","last_fullsized":"2024-03-16T20:11:46.876745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.876643+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:35.335725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.925571+0000","last_change":"2024-03-16T20:11:58.925719+0000","last_active":"2024-03-16T20:11:58.925571+0000","last_peered":"2024-03-16T20:11:58.925571+0000","last_clean":"2024-03-16T20:11:58.925571+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:11:58.925571+0000","last_undegraded":"2024-03-16T20:11:58.925571+0000","last_fullsized":"2024-03-16T20:11:58.925571+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.925363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.925363+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.925363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:39.387673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.875844+0000","last_change":"2024-03-16T20:11:50.875881+0000","last_active":"2024-03-16T20:11:50.875844+0000","last_peered":"2024-03-16T20:11:50.875844+0000","last_clean":"2024-03-16T20:11:50.875844+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:11:50.875844+0000","last_undegraded":"2024-03-16T20:11:50.875844+0000","last_fullsized":"2024-03-16T20:11:50.875844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.875738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.875738+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.875738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:08.812988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"32'5","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.302588+0000","last_change":"2024-03-16T20:11:53.302635+0000","last_active":"2024-03-16T20:11:53.302588+0000","last_peered":"2024-03-16T20:11:53.302588+0000","last_clean":"2024-03-16T20:11:53.302588+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:11:53.302588+0000","last_undegraded":"2024-03-16T20:11:53.302588+0000","last_fullsized":"2024-03-16T20:11:53.302588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:53.302493+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:53.302493+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.302493+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:55.831055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.908287+0000","last_change":"2024-03-16T20:11:57.908329+0000","last_active":"2024-03-16T20:11:57.908287+0000","last_peered":"2024-03-16T20:11:57.908287+0000","last_clean":"2024-03-16T20:11:57.908287+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:11:57.908287+0000","last_undegraded":"2024-03-16T20:11:57.908287+0000","last_fullsized":"2024-03-16T20:11:57.908287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.908177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.908177+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.908177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:49:28.348900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.898622+0000","last_change":"2024-03-16T20:12:03.898661+0000","last_active":"2024-03-16T20:12:03.898622+0000","last_peered":"2024-03-16T20:12:03.898622+0000","last_clean":"2024-03-16T20:12:03.898622+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:12:03.898622+0000","last_undegraded":"2024-03-16T20:12:03.898622+0000","last_fullsized":"2024-03-16T20:12:03.898622+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.898514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.898514+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.898514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:55.718023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.241988+0000","last_change":"2024-03-16T20:11:51.242042+0000","last_active":"2024-03-16T20:11:51.241988+0000","last_peered":"2024-03-16T20:11:51.241988+0000","last_clean":"2024-03-16T20:11:51.241988+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:11:51.241988+0000","last_undegraded":"2024-03-16T20:11:51.241988+0000","last_fullsized":"2024-03-16T20:11:51.241988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:51.241901+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:51.241901+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.241901+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:53.770463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'119","reported_seq":209,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.276981+0000","last_change":"2024-03-16T20:11:52.276981+0000","last_active":"2024-03-16T20:11:52.276981+0000","last_peered":"2024-03-16T20:11:52.276981+0000","last_clean":"2024-03-16T20:11:52.276981+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:11:52.276981+0000","last_undegraded":"2024-03-16T20:11:52.276981+0000","last_fullsized":"2024-03-16T20:11:52.276981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'119","last_scrub_stamp":"2024-03-16T20:11:52.276821+0000","last_deep_scrub":"32'119","last_deep_scrub_stamp":"2024-03-16T20:11:52.276821+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.276821+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:58.972679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.191201+0000","last_change":"2024-03-16T20:12:02.191243+0000","last_active":"2024-03-16T20:12:02.191201+0000","last_peered":"2024-03-16T20:12:02.191201+0000","last_clean":"2024-03-16T20:12:02.191201+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:12:02.191201+0000","last_undegraded":"2024-03-16T20:12:02.191201+0000","last_fullsized":"2024-03-16T20:12:02.191201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.191077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.191077+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.191077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:49.309458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:10:09.413611+0000","last_change":"2024-03-16T19:56:36.298986+0000","last_active":"2024-03-16T20:10:09.413611+0000","last_peered":"2024-03-16T20:10:09.413611+0000","last_clean":"2024-03-16T20:10:09.413611+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:10:09.413611+0000","last_undegraded":"2024-03-16T20:10:09.413611+0000","last_fullsized":"2024-03-16T20:10:09.413611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.907426+0000","last_change":"2024-03-16T20:11:47.907475+0000","last_active":"2024-03-16T20:11:47.907426+0000","last_peered":"2024-03-16T20:11:47.907426+0000","last_clean":"2024-03-16T20:11:47.907426+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:11:47.907426+0000","last_undegraded":"2024-03-16T20:11:47.907426+0000","last_fullsized":"2024-03-16T20:11:47.907426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.907316+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:21:14.912976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.177252+0000","last_change":"2024-03-16T20:12:01.177292+0000","last_active":"2024-03-16T20:12:01.177252+0000","last_peered":"2024-03-16T20:12:01.177252+0000","last_clean":"2024-03-16T20:12:01.177252+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:12:01.177252+0000","last_undegraded":"2024-03-16T20:12:01.177252+0000","last_fullsized":"2024-03-16T20:12:01.177252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.177140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.177140+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.177140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:09.390700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.186389+0000","last_change":"2024-03-16T20:11:56.186426+0000","last_active":"2024-03-16T20:11:56.186389+0000","last_peered":"2024-03-16T20:11:56.186389+0000","last_clean":"2024-03-16T20:11:56.186389+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:11:56.186389+0000","last_undegraded":"2024-03-16T20:11:56.186389+0000","last_fullsized":"2024-03-16T20:11:56.186389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.186272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.186272+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.186272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:47.836647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"32'4","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.993849+0000","last_change":"2024-03-16T20:11:44.993895+0000","last_active":"2024-03-16T20:11:44.993849+0000","last_peered":"2024-03-16T20:11:44.993849+0000","last_clean":"2024-03-16T20:11:44.993849+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:11:44.993849+0000","last_undegraded":"2024-03-16T20:11:44.993849+0000","last_fullsized":"2024-03-16T20:11:44.993849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4","last_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_deep_scrub":"32'4","last_deep_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.993741+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:31:57.200016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.885921+0000","last_change":"2024-03-16T20:11:53.885962+0000","last_active":"2024-03-16T20:11:53.885921+0000","last_peered":"2024-03-16T20:11:53.885921+0000","last_clean":"2024-03-16T20:11:53.885921+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:11:53.885921+0000","last_undegraded":"2024-03-16T20:11:53.885921+0000","last_fullsized":"2024-03-16T20:11:53.885921+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.885809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.885809+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.885809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:08.690092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.418411+0000","last_change":"2024-03-16T20:12:02.418462+0000","last_active":"2024-03-16T20:12:02.418411+0000","last_peered":"2024-03-16T20:12:02.418411+0000","last_clean":"2024-03-16T20:12:02.418411+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:12:02.418411+0000","last_undegraded":"2024-03-16T20:12:02.418411+0000","last_fullsized":"2024-03-16T20:12:02.418411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.418214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.418214+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.418214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:27.394802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"32'7","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.836407+0000","last_change":"2024-03-16T20:11:44.836458+0000","last_active":"2024-03-16T20:11:44.836407+0000","last_peered":"2024-03-16T20:11:44.836407+0000","last_clean":"2024-03-16T20:11:44.836407+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:11:44.836407+0000","last_undegraded":"2024-03-16T20:11:44.836407+0000","last_fullsized":"2024-03-16T20:11:44.836407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.836292+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:01.699700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.346871+0000","last_change":"2024-03-16T19:56:43.492074+0000","last_active":"2024-03-16T20:11:47.346871+0000","last_peered":"2024-03-16T20:11:47.346871+0000","last_clean":"2024-03-16T20:11:47.346871+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:11:47.346871+0000","last_undegraded":"2024-03-16T20:11:47.346871+0000","last_fullsized":"2024-03-16T20:11:47.346871+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:41.571007+0000","last_clean_scrub_stamp":"2024-03-16T19:56:41.571007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.418114+0000","last_change":"2024-03-16T20:12:01.418170+0000","last_active":"2024-03-16T20:12:01.418114+0000","last_peered":"2024-03-16T20:12:01.418114+0000","last_clean":"2024-03-16T20:12:01.418114+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:12:01.418114+0000","last_undegraded":"2024-03-16T20:12:01.418114+0000","last_fullsized":"2024-03-16T20:12:01.418114+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.417940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.417940+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.417940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:48.536321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"32'2","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.048733+0000","last_change":"2024-03-16T20:11:47.048781+0000","last_active":"2024-03-16T20:11:47.048733+0000","last_peered":"2024-03-16T20:11:47.048733+0000","last_clean":"2024-03-16T20:11:47.048733+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:11:47.048733+0000","last_undegraded":"2024-03-16T20:11:47.048733+0000","last_fullsized":"2024-03-16T20:11:47.048733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-03-16T20:11:47.048629+0000","last_deep_scrub":"32'2","last_deep_scrub_stamp":"2024-03-16T20:11:47.048629+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.048629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:35.507712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.949114+0000","last_change":"2024-03-16T20:12:00.949155+0000","last_active":"2024-03-16T20:12:00.949114+0000","last_peered":"2024-03-16T20:12:00.949114+0000","last_clean":"2024-03-16T20:12:00.949114+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:12:00.949114+0000","last_undegraded":"2024-03-16T20:12:00.949114+0000","last_fullsized":"2024-03-16T20:12:00.949114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.948999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.948999+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.948999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:15.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.431977+0000","last_change":"2024-03-16T20:12:03.432016+0000","last_active":"2024-03-16T20:12:03.431977+0000","last_peered":"2024-03-16T20:12:03.431977+0000","last_clean":"2024-03-16T20:12:03.431977+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:12:03.431977+0000","last_undegraded":"2024-03-16T20:12:03.431977+0000","last_fullsized":"2024-03-16T20:12:03.431977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.431867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.431867+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.431867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:29.674096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'26","reported_seq":77,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.873458+0000","last_change":"2024-03-16T20:11:45.873518+0000","last_active":"2024-03-16T20:11:45.873458+0000","last_peered":"2024-03-16T20:11:45.873458+0000","last_clean":"2024-03-16T20:11:45.873458+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:11:45.873458+0000","last_undegraded":"2024-03-16T20:11:45.873458+0000","last_fullsized":"2024-03-16T20:11:45.873458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'26","last_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_deep_scrub":"32'26","last_deep_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.873355+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:46.491674+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.855277+0000","last_change":"2024-03-16T20:11:54.855338+0000","last_active":"2024-03-16T20:11:54.855277+0000","last_peered":"2024-03-16T20:11:54.855277+0000","last_clean":"2024-03-16T20:11:54.855277+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:11:54.855277+0000","last_undegraded":"2024-03-16T20:11:54.855277+0000","last_fullsized":"2024-03-16T20:11:54.855277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.855076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.855076+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.855076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:03.068636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.218998+0000","last_change":"2024-03-16T20:11:58.219034+0000","last_active":"2024-03-16T20:11:58.218998+0000","last_peered":"2024-03-16T20:11:58.218998+0000","last_clean":"2024-03-16T20:11:58.218998+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:11:58.218998+0000","last_undegraded":"2024-03-16T20:11:58.218998+0000","last_fullsized":"2024-03-16T20:11:58.218998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.218885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.218885+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.218885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:28.085832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'9","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.170572+0000","last_change":"2024-03-16T20:11:52.170620+0000","last_active":"2024-03-16T20:11:52.170572+0000","last_peered":"2024-03-16T20:11:52.170572+0000","last_clean":"2024-03-16T20:11:52.170572+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:11:52.170572+0000","last_undegraded":"2024-03-16T20:11:52.170572+0000","last_fullsized":"2024-03-16T20:11:52.170572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'9","last_scrub_stamp":"2024-03-16T20:11:52.170475+0000","last_deep_scrub":"32'9","last_deep_scrub_stamp":"2024-03-16T20:11:52.170475+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.170475+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:18.620366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.856029+0000","last_change":"2024-03-16T20:11:58.856070+0000","last_active":"2024-03-16T20:11:58.856029+0000","last_peered":"2024-03-16T20:11:58.856029+0000","last_clean":"2024-03-16T20:11:58.856029+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:11:58.856029+0000","last_undegraded":"2024-03-16T20:11:58.856029+0000","last_fullsized":"2024-03-16T20:11:58.856029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.855920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.855920+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.855920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:08:03.774971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.382315+0000","last_change":"2024-03-16T20:12:00.382369+0000","last_active":"2024-03-16T20:12:00.382315+0000","last_peered":"2024-03-16T20:12:00.382315+0000","last_clean":"2024-03-16T20:12:00.382315+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:12:00.382315+0000","last_undegraded":"2024-03-16T20:12:00.382315+0000","last_fullsized":"2024-03-16T20:12:00.382315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.382140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.382140+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.382140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:08.595700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"32'8","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.271165+0000","last_change":"2024-03-16T20:11:50.271224+0000","last_active":"2024-03-16T20:11:50.271165+0000","last_peered":"2024-03-16T20:11:50.271165+0000","last_clean":"2024-03-16T20:11:50.271165+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:11:50.271165+0000","last_undegraded":"2024-03-16T20:11:50.271165+0000","last_fullsized":"2024-03-16T20:11:50.271165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:50.271033+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:50.271033+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.271033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:56.966891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.854296+0000","last_change":"2024-03-16T20:11:59.854336+0000","last_active":"2024-03-16T20:11:59.854296+0000","last_peered":"2024-03-16T20:11:59.854296+0000","last_clean":"2024-03-16T20:11:59.854296+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:11:59.854296+0000","last_undegraded":"2024-03-16T20:11:59.854296+0000","last_fullsized":"2024-03-16T20:11:59.854296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.854176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.854176+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.854176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:12:13.481841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.842717+0000","last_change":"2024-03-16T20:11:52.842769+0000","last_active":"2024-03-16T20:11:52.842717+0000","last_peered":"2024-03-16T20:11:52.842717+0000","last_clean":"2024-03-16T20:11:52.842717+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:11:52.842717+0000","last_undegraded":"2024-03-16T20:11:52.842717+0000","last_fullsized":"2024-03-16T20:11:52.842717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.842542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.842542+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.842542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:12.552194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"32'15","reported_seq":65,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.686996+0000","last_change":"2024-03-16T20:11:43.687053+0000","last_active":"2024-03-16T20:11:43.686996+0000","last_peered":"2024-03-16T20:11:43.686996+0000","last_clean":"2024-03-16T20:11:43.686996+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:11:43.686996+0000","last_undegraded":"2024-03-16T20:11:43.686996+0000","last_fullsized":"2024-03-16T20:11:43.686996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'15","last_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_deep_scrub":"32'15","last_deep_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.686892+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:16.356150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:04.217553+0000","last_change":"2024-03-16T20:12:04.217601+0000","last_active":"2024-03-16T20:12:04.217553+0000","last_peered":"2024-03-16T20:12:04.217553+0000","last_clean":"2024-03-16T20:12:04.217553+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:12:04.217553+0000","last_undegraded":"2024-03-16T20:12:04.217553+0000","last_fullsized":"2024-03-16T20:12:04.217553+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:04.217414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:04.217414+0000","last_clean_scrub_stamp":"2024-03-16T20:12:04.217414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:12:17.845079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.402127+0000","last_change":"2024-03-16T20:11:59.402179+0000","last_active":"2024-03-16T20:11:59.402127+0000","last_peered":"2024-03-16T20:11:59.402127+0000","last_clean":"2024-03-16T20:11:59.402127+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:11:59.402127+0000","last_undegraded":"2024-03-16T20:11:59.402127+0000","last_fullsized":"2024-03-16T20:11:59.402127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.401946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.401946+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.401946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:14.206230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"32'20","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.222202+0000","last_change":"2024-03-16T20:11:49.222234+0000","last_active":"2024-03-16T20:11:49.222202+0000","last_peered":"2024-03-16T20:11:49.222202+0000","last_clean":"2024-03-16T20:11:49.222202+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:11:49.222202+0000","last_undegraded":"2024-03-16T20:11:49.222202+0000","last_fullsized":"2024-03-16T20:11:49.222202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:49.222140+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:49.222140+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.222140+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:42:28.648561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.004808+0000","last_change":"2024-03-16T20:11:56.004850+0000","last_active":"2024-03-16T20:11:56.004808+0000","last_peered":"2024-03-16T20:11:56.004808+0000","last_clean":"2024-03-16T20:11:56.004808+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:11:56.004808+0000","last_undegraded":"2024-03-16T20:11:56.004808+0000","last_fullsized":"2024-03-16T20:11:56.004808+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.004687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.004687+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.004687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:13:10.810213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.985844+0000","last_change":"2024-03-16T20:11:48.985887+0000","last_active":"2024-03-16T20:11:48.985844+0000","last_peered":"2024-03-16T20:11:48.985844+0000","last_clean":"2024-03-16T20:11:48.985844+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:11:48.985844+0000","last_undegraded":"2024-03-16T20:11:48.985844+0000","last_fullsized":"2024-03-16T20:11:48.985844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.985737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.985737+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.985737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:15.149371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"32'9","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.098421+0000","last_change":"2024-03-16T20:11:50.098471+0000","last_active":"2024-03-16T20:11:50.098421+0000","last_peered":"2024-03-16T20:11:50.098421+0000","last_clean":"2024-03-16T20:11:50.098421+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:11:50.098421+0000","last_undegraded":"2024-03-16T20:11:50.098421+0000","last_fullsized":"2024-03-16T20:11:50.098421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'9","last_scrub_stamp":"2024-03-16T20:11:50.098317+0000","last_deep_scrub":"32'9","last_deep_scrub_stamp":"2024-03-16T20:11:50.098317+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.098317+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:50.425995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.986198+0000","last_change":"2024-03-16T20:11:53.986252+0000","last_active":"2024-03-16T20:11:53.986198+0000","last_peered":"2024-03-16T20:11:53.986198+0000","last_clean":"2024-03-16T20:11:53.986198+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:11:53.986198+0000","last_undegraded":"2024-03-16T20:11:53.986198+0000","last_fullsized":"2024-03-16T20:11:53.986198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.986036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.986036+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.986036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:00.681893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.766369+0000","last_change":"2024-03-16T20:11:45.766407+0000","last_active":"2024-03-16T20:11:45.766369+0000","last_peered":"2024-03-16T20:11:45.766369+0000","last_clean":"2024-03-16T20:11:45.766369+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:11:45.766369+0000","last_undegraded":"2024-03-16T20:11:45.766369+0000","last_fullsized":"2024-03-16T20:11:45.766369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.766246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:05.475898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"32'6","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.216678+0000","last_change":"2024-03-16T20:11:54.216728+0000","last_active":"2024-03-16T20:11:54.216678+0000","last_peered":"2024-03-16T20:11:54.216678+0000","last_clean":"2024-03-16T20:11:54.216678+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:11:54.216678+0000","last_undegraded":"2024-03-16T20:11:54.216678+0000","last_fullsized":"2024-03-16T20:11:54.216678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-03-16T20:11:54.216566+0000","last_deep_scrub":"32'6","last_deep_scrub_stamp":"2024-03-16T20:11:54.216566+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.216566+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:01:09.020994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:04.862515+0000","last_change":"2024-03-16T20:12:04.862554+0000","last_active":"2024-03-16T20:12:04.862515+0000","last_peered":"2024-03-16T20:12:04.862515+0000","last_clean":"2024-03-16T20:12:04.862515+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:12:04.862515+0000","last_undegraded":"2024-03-16T20:12:04.862515+0000","last_fullsized":"2024-03-16T20:12:04.862515+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:04.862404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:04.862404+0000","last_clean_scrub_stamp":"2024-03-16T20:12:04.862404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:12.258860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.141416+0000","last_change":"2024-03-16T20:12:01.141536+0000","last_active":"2024-03-16T20:12:01.141416+0000","last_peered":"2024-03-16T20:12:01.141416+0000","last_clean":"2024-03-16T20:12:01.141416+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:12:01.141416+0000","last_undegraded":"2024-03-16T20:12:01.141416+0000","last_fullsized":"2024-03-16T20:12:01.141416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.141112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.141112+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.141112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:08:42.178461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"32'7","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.134000+0000","last_change":"2024-03-16T20:11:53.134050+0000","last_active":"2024-03-16T20:11:53.134000+0000","last_peered":"2024-03-16T20:11:53.134000+0000","last_clean":"2024-03-16T20:11:53.134000+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:11:53.134000+0000","last_undegraded":"2024-03-16T20:11:53.134000+0000","last_fullsized":"2024-03-16T20:11:53.134000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:53.133903+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:53.133903+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.133903+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:54:11.540926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.866641+0000","last_change":"2024-03-16T20:12:01.866681+0000","last_active":"2024-03-16T20:12:01.866641+0000","last_peered":"2024-03-16T20:12:01.866641+0000","last_clean":"2024-03-16T20:12:01.866641+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:12:01.866641+0000","last_undegraded":"2024-03-16T20:12:01.866641+0000","last_fullsized":"2024-03-16T20:12:01.866641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.866518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.866518+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.866518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:51:13.427578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.863668+0000","last_change":"2024-03-16T20:11:56.863707+0000","last_active":"2024-03-16T20:11:56.863668+0000","last_peered":"2024-03-16T20:11:56.863668+0000","last_clean":"2024-03-16T20:11:56.863668+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:11:56.863668+0000","last_undegraded":"2024-03-16T20:11:56.863668+0000","last_fullsized":"2024-03-16T20:11:56.863668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.863556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.863556+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.863556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:21.840144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.082943+0000","last_change":"2024-03-16T20:11:51.082979+0000","last_active":"2024-03-16T20:11:51.082943+0000","last_peered":"2024-03-16T20:11:51.082943+0000","last_clean":"2024-03-16T20:11:51.082943+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:11:51.082943+0000","last_undegraded":"2024-03-16T20:11:51.082943+0000","last_fullsized":"2024-03-16T20:11:51.082943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.082823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.082823+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.082823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:06.361861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"32'4","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.213556+0000","last_change":"2024-03-16T20:11:53.213671+0000","last_active":"2024-03-16T20:11:53.213556+0000","last_peered":"2024-03-16T20:11:53.213556+0000","last_clean":"2024-03-16T20:11:53.213556+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:11:53.213556+0000","last_undegraded":"2024-03-16T20:11:53.213556+0000","last_fullsized":"2024-03-16T20:11:53.213556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4","last_scrub_stamp":"2024-03-16T20:11:53.213289+0000","last_deep_scrub":"32'4","last_deep_scrub_stamp":"2024-03-16T20:11:53.213289+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.213289+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:54.080243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.170593+0000","last_change":"2024-03-16T20:12:03.170618+0000","last_active":"2024-03-16T20:12:03.170593+0000","last_peered":"2024-03-16T20:12:03.170593+0000","last_clean":"2024-03-16T20:12:03.170593+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:12:03.170593+0000","last_undegraded":"2024-03-16T20:12:03.170593+0000","last_fullsized":"2024-03-16T20:12:03.170593+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.170526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.170526+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.170526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:28.147113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.883135+0000","last_change":"2024-03-16T20:11:45.883182+0000","last_active":"2024-03-16T20:11:45.883135+0000","last_peered":"2024-03-16T20:11:45.883135+0000","last_clean":"2024-03-16T20:11:45.883135+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:11:45.883135+0000","last_undegraded":"2024-03-16T20:11:45.883135+0000","last_fullsized":"2024-03-16T20:11:45.883135+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.883021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:48.411583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"32'5","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.878274+0000","last_change":"2024-03-16T20:11:51.878321+0000","last_active":"2024-03-16T20:11:51.878274+0000","last_peered":"2024-03-16T20:11:51.878274+0000","last_clean":"2024-03-16T20:11:51.878274+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:11:51.878274+0000","last_undegraded":"2024-03-16T20:11:51.878274+0000","last_fullsized":"2024-03-16T20:11:51.878274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:51.878180+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:51.878180+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.878180+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:58:33.086060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"32'10","reported_seq":60,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.147500+0000","last_change":"2024-03-16T20:11:51.147557+0000","last_active":"2024-03-16T20:11:51.147500+0000","last_peered":"2024-03-16T20:11:51.147500+0000","last_clean":"2024-03-16T20:11:51.147500+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:11:51.147500+0000","last_undegraded":"2024-03-16T20:11:51.147500+0000","last_fullsized":"2024-03-16T20:11:51.147500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'10","last_scrub_stamp":"2024-03-16T20:11:51.147398+0000","last_deep_scrub":"32'10","last_deep_scrub_stamp":"2024-03-16T20:11:51.147398+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.147398+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:56.611534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.819175+0000","last_change":"2024-03-16T20:11:47.819214+0000","last_active":"2024-03-16T20:11:47.819175+0000","last_peered":"2024-03-16T20:11:47.819175+0000","last_clean":"2024-03-16T20:11:47.819175+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:11:47.819175+0000","last_undegraded":"2024-03-16T20:11:47.819175+0000","last_fullsized":"2024-03-16T20:11:47.819175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.819062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:50.347624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.818673+0000","last_change":"2024-03-16T20:11:50.818792+0000","last_active":"2024-03-16T20:11:50.818673+0000","last_peered":"2024-03-16T20:11:50.818673+0000","last_clean":"2024-03-16T20:11:50.818673+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:11:50.818673+0000","last_undegraded":"2024-03-16T20:11:50.818673+0000","last_fullsized":"2024-03-16T20:11:50.818673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.818443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.818443+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.818443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:31.275107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"32'7","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.850558+0000","last_change":"2024-03-16T20:11:49.850605+0000","last_active":"2024-03-16T20:11:49.850558+0000","last_peered":"2024-03-16T20:11:49.850558+0000","last_clean":"2024-03-16T20:11:49.850558+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:11:49.850558+0000","last_undegraded":"2024-03-16T20:11:49.850558+0000","last_fullsized":"2024-03-16T20:11:49.850558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:49.850461+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:49.850461+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.850461+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:19.717407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.023506+0000","last_change":"2024-03-16T20:11:48.023544+0000","last_active":"2024-03-16T20:11:48.023506+0000","last_peered":"2024-03-16T20:11:48.023506+0000","last_clean":"2024-03-16T20:11:48.023506+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:11:48.023506+0000","last_undegraded":"2024-03-16T20:11:48.023506+0000","last_fullsized":"2024-03-16T20:11:48.023506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.023400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.023400+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.023400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:17.327792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.870788+0000","last_change":"2024-03-16T20:11:57.870832+0000","last_active":"2024-03-16T20:11:57.870788+0000","last_peered":"2024-03-16T20:11:57.870788+0000","last_clean":"2024-03-16T20:11:57.870788+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:11:57.870788+0000","last_undegraded":"2024-03-16T20:11:57.870788+0000","last_fullsized":"2024-03-16T20:11:57.870788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.870655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.870655+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.870655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:56.093848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.212579+0000","last_change":"2024-03-16T20:12:00.212616+0000","last_active":"2024-03-16T20:12:00.212579+0000","last_peered":"2024-03-16T20:12:00.212579+0000","last_clean":"2024-03-16T20:12:00.212579+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:12:00.212579+0000","last_undegraded":"2024-03-16T20:12:00.212579+0000","last_fullsized":"2024-03-16T20:12:00.212579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.212475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.212475+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.212475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:58:41.420354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.876522+0000","last_change":"2024-03-16T20:11:48.876540+0000","last_active":"2024-03-16T20:11:48.876522+0000","last_peered":"2024-03-16T20:11:48.876522+0000","last_clean":"2024-03-16T20:11:48.876522+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:11:48.876522+0000","last_undegraded":"2024-03-16T20:11:48.876522+0000","last_fullsized":"2024-03-16T20:11:48.876522+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.876460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.876460+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.876460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:15.040094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.277449+0000","last_change":"2024-03-16T20:11:49.277493+0000","last_active":"2024-03-16T20:11:49.277449+0000","last_peered":"2024-03-16T20:11:49.277449+0000","last_clean":"2024-03-16T20:11:49.277449+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:11:49.277449+0000","last_undegraded":"2024-03-16T20:11:49.277449+0000","last_fullsized":"2024-03-16T20:11:49.277449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.277321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.277321+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.277321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:08.986973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.873763+0000","last_change":"2024-03-16T20:11:51.873803+0000","last_active":"2024-03-16T20:11:51.873763+0000","last_peered":"2024-03-16T20:11:51.873763+0000","last_clean":"2024-03-16T20:11:51.873763+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:11:51.873763+0000","last_undegraded":"2024-03-16T20:11:51.873763+0000","last_fullsized":"2024-03-16T20:11:51.873763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.873657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.873657+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.873657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:21.740603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.877090+0000","last_change":"2024-03-16T20:11:53.877131+0000","last_active":"2024-03-16T20:11:53.877090+0000","last_peered":"2024-03-16T20:11:53.877090+0000","last_clean":"2024-03-16T20:11:53.877090+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:11:53.877090+0000","last_undegraded":"2024-03-16T20:11:53.877090+0000","last_fullsized":"2024-03-16T20:11:53.877090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.876983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.876983+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.876983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:40.995365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.058999+0000","last_change":"2024-03-16T20:11:52.059035+0000","last_active":"2024-03-16T20:11:52.058999+0000","last_peered":"2024-03-16T20:11:52.058999+0000","last_clean":"2024-03-16T20:11:52.058999+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:11:52.058999+0000","last_undegraded":"2024-03-16T20:11:52.058999+0000","last_fullsized":"2024-03-16T20:11:52.058999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.058883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.058883+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.058883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:23.372950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.178239+0000","last_change":"2024-03-16T20:11:59.178344+0000","last_active":"2024-03-16T20:11:59.178239+0000","last_peered":"2024-03-16T20:11:59.178239+0000","last_clean":"2024-03-16T20:11:59.178239+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:11:59.178239+0000","last_undegraded":"2024-03-16T20:11:59.178239+0000","last_fullsized":"2024-03-16T20:11:59.178239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.177970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.177970+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.177970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:14.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.858816+0000","last_change":"2024-03-16T20:11:54.858910+0000","last_active":"2024-03-16T20:11:54.858816+0000","last_peered":"2024-03-16T20:11:54.858816+0000","last_clean":"2024-03-16T20:11:54.858816+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:11:54.858816+0000","last_undegraded":"2024-03-16T20:11:54.858816+0000","last_fullsized":"2024-03-16T20:11:54.858816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.858562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.858562+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.858562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:19.835150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.181761+0000","last_change":"2024-03-16T20:11:55.181799+0000","last_active":"2024-03-16T20:11:55.181761+0000","last_peered":"2024-03-16T20:11:55.181761+0000","last_clean":"2024-03-16T20:11:55.181761+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:11:55.181761+0000","last_undegraded":"2024-03-16T20:11:55.181761+0000","last_fullsized":"2024-03-16T20:11:55.181761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.181654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.181654+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.181654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:47.930472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.897698+0000","last_change":"2024-03-16T20:11:55.897835+0000","last_active":"2024-03-16T20:11:55.897698+0000","last_peered":"2024-03-16T20:11:55.897698+0000","last_clean":"2024-03-16T20:11:55.897698+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:11:55.897698+0000","last_undegraded":"2024-03-16T20:11:55.897698+0000","last_fullsized":"2024-03-16T20:11:55.897698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.897455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.897455+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.897455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:22.139684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.166051+0000","last_change":"2024-03-16T20:12:00.166087+0000","last_active":"2024-03-16T20:12:00.166051+0000","last_peered":"2024-03-16T20:12:00.166051+0000","last_clean":"2024-03-16T20:12:00.166051+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:12:00.166051+0000","last_undegraded":"2024-03-16T20:12:00.166051+0000","last_fullsized":"2024-03-16T20:12:00.166051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.165933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.165933+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.165933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:26.408162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.867223+0000","last_change":"2024-03-16T20:11:52.867262+0000","last_active":"2024-03-16T20:11:52.867223+0000","last_peered":"2024-03-16T20:11:52.867223+0000","last_clean":"2024-03-16T20:11:52.867223+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:11:52.867223+0000","last_undegraded":"2024-03-16T20:11:52.867223+0000","last_fullsized":"2024-03-16T20:11:52.867223+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.867111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.867111+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.867111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:08.289839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.145498+0000","last_change":"2024-03-16T20:11:56.145536+0000","last_active":"2024-03-16T20:11:56.145498+0000","last_peered":"2024-03-16T20:11:56.145498+0000","last_clean":"2024-03-16T20:11:56.145498+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:11:56.145498+0000","last_undegraded":"2024-03-16T20:11:56.145498+0000","last_fullsized":"2024-03-16T20:11:56.145498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.145390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.145390+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.145390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:51:51.466788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.275184+0000","last_change":"2024-03-16T20:11:50.275299+0000","last_active":"2024-03-16T20:11:50.275184+0000","last_peered":"2024-03-16T20:11:50.275184+0000","last_clean":"2024-03-16T20:11:50.275184+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:11:50.275184+0000","last_undegraded":"2024-03-16T20:11:50.275184+0000","last_fullsized":"2024-03-16T20:11:50.275184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.274977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.274977+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.274977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:55.739112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.849641+0000","last_change":"2024-03-16T20:11:49.849679+0000","last_active":"2024-03-16T20:11:49.849641+0000","last_peered":"2024-03-16T20:11:49.849641+0000","last_clean":"2024-03-16T20:11:49.849641+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:11:49.849641+0000","last_undegraded":"2024-03-16T20:11:49.849641+0000","last_fullsized":"2024-03-16T20:11:49.849641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.849534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.849534+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.849534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:12:20.595436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.183023+0000","last_change":"2024-03-16T20:12:03.183057+0000","last_active":"2024-03-16T20:12:03.183023+0000","last_peered":"2024-03-16T20:12:03.183023+0000","last_clean":"2024-03-16T20:12:03.183023+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:12:03.183023+0000","last_undegraded":"2024-03-16T20:12:03.183023+0000","last_fullsized":"2024-03-16T20:12:03.183023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.182944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.182944+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.182944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:56:14.282067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.178440+0000","last_change":"2024-03-16T20:11:59.178476+0000","last_active":"2024-03-16T20:11:59.178440+0000","last_peered":"2024-03-16T20:11:59.178440+0000","last_clean":"2024-03-16T20:11:59.178440+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:11:59.178440+0000","last_undegraded":"2024-03-16T20:11:59.178440+0000","last_fullsized":"2024-03-16T20:11:59.178440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.178344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.178344+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.178344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:24.154931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.058485+0000","last_change":"2024-03-16T20:11:54.058540+0000","last_active":"2024-03-16T20:11:54.058485+0000","last_peered":"2024-03-16T20:11:54.058485+0000","last_clean":"2024-03-16T20:11:54.058485+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:11:54.058485+0000","last_undegraded":"2024-03-16T20:11:54.058485+0000","last_fullsized":"2024-03-16T20:11:54.058485+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.058308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.058308+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.058308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:58.415808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.834025+0000","last_change":"2024-03-16T20:11:49.834066+0000","last_active":"2024-03-16T20:11:49.834025+0000","last_peered":"2024-03-16T20:11:49.834025+0000","last_clean":"2024-03-16T20:11:49.834025+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:11:49.834025+0000","last_undegraded":"2024-03-16T20:11:49.834025+0000","last_fullsized":"2024-03-16T20:11:49.834025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.833918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.833918+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.833918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:50.700873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.864092+0000","last_change":"2024-03-16T20:11:56.864132+0000","last_active":"2024-03-16T20:11:56.864092+0000","last_peered":"2024-03-16T20:11:56.864092+0000","last_clean":"2024-03-16T20:11:56.864092+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:11:56.864092+0000","last_undegraded":"2024-03-16T20:11:56.864092+0000","last_fullsized":"2024-03-16T20:11:56.864092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.863991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.863991+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.863991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:08:37.901340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.977309+0000","last_change":"2024-03-16T20:11:51.977345+0000","last_active":"2024-03-16T20:11:51.977309+0000","last_peered":"2024-03-16T20:11:51.977309+0000","last_clean":"2024-03-16T20:11:51.977309+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:11:51.977309+0000","last_undegraded":"2024-03-16T20:11:51.977309+0000","last_fullsized":"2024-03-16T20:11:51.977309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.977190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.977190+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.977190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:13.226901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.161201+0000","last_change":"2024-03-16T20:11:54.161238+0000","last_active":"2024-03-16T20:11:54.161201+0000","last_peered":"2024-03-16T20:11:54.161201+0000","last_clean":"2024-03-16T20:11:54.161201+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:11:54.161201+0000","last_undegraded":"2024-03-16T20:11:54.161201+0000","last_fullsized":"2024-03-16T20:11:54.161201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.161095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.161095+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.161095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:11:06.244614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.188449+0000","last_change":"2024-03-16T20:11:57.188489+0000","last_active":"2024-03-16T20:11:57.188449+0000","last_peered":"2024-03-16T20:11:57.188449+0000","last_clean":"2024-03-16T20:11:57.188449+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:11:57.188449+0000","last_undegraded":"2024-03-16T20:11:57.188449+0000","last_fullsized":"2024-03-16T20:11:57.188449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.188342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.188342+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.188342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:10.191861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:14.305594+0000","last_change":"2024-03-16T19:56:36.299588+0000","last_active":"2024-03-16T20:11:14.305594+0000","last_peered":"2024-03-16T20:11:14.305594+0000","last_clean":"2024-03-16T20:11:14.305594+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:11:14.305594+0000","last_undegraded":"2024-03-16T20:11:14.305594+0000","last_fullsized":"2024-03-16T20:11:14.305594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.840694+0000","last_change":"2024-03-16T20:11:48.840737+0000","last_active":"2024-03-16T20:11:48.840694+0000","last_peered":"2024-03-16T20:11:48.840694+0000","last_clean":"2024-03-16T20:11:48.840694+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:11:48.840694+0000","last_undegraded":"2024-03-16T20:11:48.840694+0000","last_fullsized":"2024-03-16T20:11:48.840694+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.840578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.840578+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.840578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:01.844097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:39.309515+0000","last_change":"2024-03-16T19:56:36.297837+0000","last_active":"2024-03-16T20:11:39.309515+0000","last_peered":"2024-03-16T20:11:39.309515+0000","last_clean":"2024-03-16T20:11:39.309515+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:11:39.309515+0000","last_undegraded":"2024-03-16T20:11:39.309515+0000","last_fullsized":"2024-03-16T20:11:39.309515+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.189723+0000","last_change":"2024-03-16T20:12:02.189759+0000","last_active":"2024-03-16T20:12:02.189723+0000","last_peered":"2024-03-16T20:12:02.189723+0000","last_clean":"2024-03-16T20:12:02.189723+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:12:02.189723+0000","last_undegraded":"2024-03-16T20:12:02.189723+0000","last_fullsized":"2024-03-16T20:12:02.189723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.189621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.189621+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.189621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:00.412815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.301282+0000","last_change":"2024-03-16T20:11:51.301376+0000","last_active":"2024-03-16T20:11:51.301282+0000","last_peered":"2024-03-16T20:11:51.301282+0000","last_clean":"2024-03-16T20:11:51.301282+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:11:51.301282+0000","last_undegraded":"2024-03-16T20:11:51.301282+0000","last_fullsized":"2024-03-16T20:11:51.301282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.301088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.301088+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.301088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:02:19.083559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.797896+0000","last_change":"2024-03-16T20:11:51.797931+0000","last_active":"2024-03-16T20:11:51.797896+0000","last_peered":"2024-03-16T20:11:51.797896+0000","last_clean":"2024-03-16T20:11:51.797896+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:11:51.797896+0000","last_undegraded":"2024-03-16T20:11:51.797896+0000","last_fullsized":"2024-03-16T20:11:51.797896+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.797788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.797788+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.797788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:13.047499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.718717+0000","last_change":"2024-03-16T20:11:44.718754+0000","last_active":"2024-03-16T20:11:44.718717+0000","last_peered":"2024-03-16T20:11:44.718717+0000","last_clean":"2024-03-16T20:11:44.718717+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:11:44.718717+0000","last_undegraded":"2024-03-16T20:11:44.718717+0000","last_fullsized":"2024-03-16T20:11:44.718717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.718600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:50.182736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.015497+0000","last_change":"2024-03-16T20:11:55.015535+0000","last_active":"2024-03-16T20:11:55.015497+0000","last_peered":"2024-03-16T20:11:55.015497+0000","last_clean":"2024-03-16T20:11:55.015497+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:11:55.015497+0000","last_undegraded":"2024-03-16T20:11:55.015497+0000","last_fullsized":"2024-03-16T20:11:55.015497+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.015394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.015394+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.015394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:35.056024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:14.305579+0000","last_change":"2024-03-16T19:56:36.302238+0000","last_active":"2024-03-16T20:11:14.305579+0000","last_peered":"2024-03-16T20:11:14.305579+0000","last_clean":"2024-03-16T20:11:14.305579+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:11:14.305579+0000","last_undegraded":"2024-03-16T20:11:14.305579+0000","last_fullsized":"2024-03-16T20:11:14.305579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:56:34.722570+0000","last_clean_scrub_stamp":"2024-03-16T19:56:34.722570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476935,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476937,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476936,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476934,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509638,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509640,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509640,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542343,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:12:10.867 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:12:10.867 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-16T19:56:41.571007+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=41, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:12:10.868 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:12:10.868 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:12:10.868 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-16T19:56:34.722570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=56, tm_sec=34, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=20, tm_min=11, tm_sec=40, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T20:12:10.868 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T20:12:13.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:12:13.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:12:30.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:12:30.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:30.999+0000 7febf4fe1640 1 Processor -- start 2024-03-16T20:12:30.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:30.999+0000 7febf4fe1640 1 -- start start 2024-03-16T20:12:30.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.003+0000 7febf4fe1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 0x7febf015b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:30.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.003+0000 7febf4fe1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf015ab60 con 0x7febf01540c0 2024-03-16T20:12:30.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.003+0000 7febf4fe1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf0159fb0 con 0x7febf00c8550 2024-03-16T20:12:30.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.003+0000 7febf4fe1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf006fb80 con 0x7febf01544a0 2024-03-16T20:12:31.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.031+0000 7febee59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7febf01540c0 0x7febf006f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43138/0 (socket says 172.21.15.32:43138) 2024-03-16T20:12:31.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.031+0000 7febee59c640 1 -- 172.21.15.32:0/3031846620 learned_addr learned my addr 172.21.15.32:0/3031846620 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-16T20:12:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.035+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004075901 0 0) 0x7febf015ab60 con 0x7febf01540c0 2024-03-16T20:12:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.035+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc003530 con 0x7febf01540c0 2024-03-16T20:12:31.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2647972364 0 0) 0x7febdc003530 con 0x7febf01540c0 2024-03-16T20:12:31.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 msgr2=0x7febf015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 0x7febf015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 >> v1:172.21.15.59:6789/0 conn(0x7febf00c8550 legacy=0x7febf00cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf015a230 con 0x7febf01540c0 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3633048475 0 0) 0x7febe0001680 con 0x7febf01540c0 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 >> v1:172.21.15.32:6789/0 conn(0x7febf01540c0 legacy=0x7febf006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 0x7febdc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc0040f0 0x7febdc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febdc004e30 0x7febdc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf006fb80 con 0x7febdc004e30 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc005540 con 0x7febdc0040f0 2024-03-16T20:12:31.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc00bed0 con 0x7febf01544a0 2024-03-16T20:12:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.039+0000 7febeed9d640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febdc004e30 0x7febdc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.043+0000 7febeed9d640 1 -- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 msgr2=0x7febdc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.043+0000 7febeed9d640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf01544a0 0x7febdc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.043+0000 7febeed9d640 1 -- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc0040f0 msgr2=0x7febdc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.043+0000 7febeed9d640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc0040f0 0x7febdc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.043+0000 7febeed9d640 1 -- 172.21.15.32:0/3031846620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc003530 con 0x7febdc004e30 2024-03-16T20:12:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.047+0000 7febeed9d640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febdc004e30 0x7febdc00b990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7febf00c89e0 tx=0x7febe801c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313e8366394fe34b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8003ba0 con 0x7febdc004e30 2024-03-16T20:12:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8003d00 con 0x7febdc004e30 2024-03-16T20:12:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febedd9b640 1 -- 172.21.15.32:0/3031846620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8005290 con 0x7febdc004e30 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- 172.21.15.32:0/3031846620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febdc004e30 msgr2=0x7febdc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 --2- 172.21.15.32:0/3031846620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febdc004e30 0x7febdc00b990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7febf00c89e0 tx=0x7febe801c680 comp rx=0 tx=0).stop 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febeed9d640 1 -- 172.21.15.32:0/3031846620 reap_dead start 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- 172.21.15.32:0/3031846620 shutdown_connections 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- 172.21.15.32:0/3031846620 >> 172.21.15.32:0/3031846620 conn(0x7febf0066ea0 msgr2=0x7febf00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- 172.21.15.32:0/3031846620 shutdown_connections 2024-03-16T20:12:31.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- 172.21.15.32:0/3031846620 wait complete. 2024-03-16T20:12:31.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 Processor -- start 2024-03-16T20:12:31.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.055+0000 7febf4fe1640 1 -- start start 2024-03-16T20:12:31.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc004e30 0x7febf01f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf00c76a0 0x7febf00c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf014e400 con 0x7febf01efde0 2024-03-16T20:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01f0220 con 0x7febdc004e30 2024-03-16T20:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febf4fe1640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf01ef950 con 0x7febf00c76a0 2024-03-16T20:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febe7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febe7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55318/0 (socket says 172.21.15.32:55318) 2024-03-16T20:12:31.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.059+0000 7febe7fff640 1 -- 172.21.15.32:0/1754370965 learned_addr learned my addr 172.21.15.32:0/1754370965 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:12:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.063+0000 7febe7fff640 1 -- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf00c76a0 msgr2=0x7febf00c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.063+0000 7febe7fff640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf00c76a0 0x7febf00c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.063+0000 7febe7fff640 1 -- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc004e30 msgr2=0x7febf01f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.063+0000 7febe7fff640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc004e30 0x7febf01f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.063+0000 7febe7fff640 1 -- 172.21.15.32:0/1754370965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf01f3120 con 0x7febf01efde0 2024-03-16T20:12:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.067+0000 7febe7fff640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7febd800b480 tx=0x7febd800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.071+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8011830 con 0x7febf01efde0 2024-03-16T20:12:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.071+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8011990 con 0x7febf01efde0 2024-03-16T20:12:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.071+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8011d40 con 0x7febf01efde0 2024-03-16T20:12:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.075+0000 7febf4fe1640 1 -- 172.21.15.32:0/1754370965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf01f33e0 con 0x7febf01efde0 2024-03-16T20:12:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.075+0000 7febf4fe1640 1 -- 172.21.15.32:0/1754370965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf01ef300 con 0x7febf01efde0 2024-03-16T20:12:31.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.075+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc80052a0 con 0x7febf01efde0 2024-03-16T20:12:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.343+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7febd801a400 con 0x7febf01efde0 2024-03-16T20:12:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.343+0000 7febe6ffd640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 0x7febd0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.343+0000 7febee59c640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 0x7febd0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.347+0000 7febee59c640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 0x7febd0069380 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7febe0001ec0 tx=0x7febe0001610 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:31.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.375+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7febd800f2e0 con 0x7febf01efde0 2024-03-16T20:12:31.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.375+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7febd8051f60 con 0x7febf01efde0 2024-03-16T20:12:31.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.555+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 --> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7febc8000bd0 con 0x7febd0066ec0 2024-03-16T20:12:31.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.571+0000 7febe6ffd640 1 -- 172.21.15.32:0/1754370965 <== mgr.4102 v2:172.21.15.32:6800/1649113964 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332762 (secure 0 0 0) 0x7febc8000bd0 con 0x7febd0066ec0 2024-03-16T20:12:31.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-16T20:12:31.573 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 msgr2=0x7febd0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 0x7febd0069380 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7febe0001ec0 tx=0x7febe0001610 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 msgr2=0x7febf01f27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7febd800b480 tx=0x7febd800b940 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 shutdown_connections 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7febd0066ec0 0x7febd0069380 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7febf00c76a0 0x7febf00c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febf01efde0 0x7febf01f27e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 --2- 172.21.15.32:0/1754370965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febdc004e30 0x7febf01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 >> 172.21.15.32:0/1754370965 conn(0x7febf0066ea0 msgr2=0x7febf00672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 shutdown_connections 2024-03-16T20:12:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.575+0000 7febe4ff9640 1 -- 172.21.15.32:0/1754370965 wait complete. 2024-03-16T20:12:31.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-03-16T20:12:30.862480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":367,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241000,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733720,"statfs":{"total":773094113280,"available":772847329280,"internally_reserved":0,"allocated":4169728,"data_stored":2356864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004072"},"pg_stats":[{"pgid":"2.1f","version":"32'20","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:42.724708+0000","last_change":"2024-03-16T20:11:42.724767+0000","last_active":"2024-03-16T20:11:42.724708+0000","last_peered":"2024-03-16T20:11:42.724708+0000","last_clean":"2024-03-16T20:11:42.724708+0000","last_became_active":"2024-03-16T19:56:33.962716+0000","last_became_peered":"2024-03-16T19:56:33.962716+0000","last_unstale":"2024-03-16T20:11:42.724708+0000","last_undegraded":"2024-03-16T20:11:42.724708+0000","last_fullsized":"2024-03-16T20:11:42.724708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:42.724584+0000","last_clean_scrub_stamp":"2024-03-16T20:11:42.724584+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:20.296590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.927855+0000","last_change":"2024-03-16T20:11:46.927899+0000","last_active":"2024-03-16T20:11:46.927855+0000","last_peered":"2024-03-16T20:11:46.927855+0000","last_clean":"2024-03-16T20:11:46.927855+0000","last_became_active":"2024-03-16T19:56:36.309003+0000","last_became_peered":"2024-03-16T19:56:36.309003+0000","last_unstale":"2024-03-16T20:11:46.927855+0000","last_undegraded":"2024-03-16T20:11:46.927855+0000","last_fullsized":"2024-03-16T20:11:46.927855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.927737+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.927737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.399583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.913457+0000","last_change":"2024-03-16T20:12:02.913500+0000","last_active":"2024-03-16T20:12:02.913457+0000","last_peered":"2024-03-16T20:12:02.913457+0000","last_clean":"2024-03-16T20:12:02.913457+0000","last_became_active":"2024-03-16T19:56:43.493905+0000","last_became_peered":"2024-03-16T19:56:43.493905+0000","last_unstale":"2024-03-16T20:12:02.913457+0000","last_undegraded":"2024-03-16T20:12:02.913457+0000","last_fullsized":"2024-03-16T20:12:02.913457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.913346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.913346+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.913346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:24.865908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"32'3","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.850227+0000","last_change":"2024-03-16T20:11:50.850272+0000","last_active":"2024-03-16T20:11:50.850227+0000","last_peered":"2024-03-16T20:11:50.850227+0000","last_clean":"2024-03-16T20:11:50.850227+0000","last_became_active":"2024-03-16T19:56:33.962988+0000","last_became_peered":"2024-03-16T19:56:33.962988+0000","last_unstale":"2024-03-16T20:11:50.850227+0000","last_undegraded":"2024-03-16T20:11:50.850227+0000","last_fullsized":"2024-03-16T20:11:50.850227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3","last_scrub_stamp":"2024-03-16T20:11:50.850119+0000","last_deep_scrub":"32'3","last_deep_scrub_stamp":"2024-03-16T20:11:50.850119+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.850119+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:06.272846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.783367+0000","last_change":"2024-03-16T20:11:46.783404+0000","last_active":"2024-03-16T20:11:46.783367+0000","last_peered":"2024-03-16T20:11:46.783367+0000","last_clean":"2024-03-16T20:11:46.783367+0000","last_became_active":"2024-03-16T19:56:36.303598+0000","last_became_peered":"2024-03-16T19:56:36.303598+0000","last_unstale":"2024-03-16T20:11:46.783367+0000","last_undegraded":"2024-03-16T20:11:46.783367+0000","last_fullsized":"2024-03-16T20:11:46.783367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:46.783256+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.783256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:53.479114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.996122+0000","last_change":"2024-03-16T20:11:54.996183+0000","last_active":"2024-03-16T20:11:54.996122+0000","last_peered":"2024-03-16T20:11:54.996122+0000","last_clean":"2024-03-16T20:11:54.996122+0000","last_became_active":"2024-03-16T19:56:43.457185+0000","last_became_peered":"2024-03-16T19:56:43.457185+0000","last_unstale":"2024-03-16T20:11:54.996122+0000","last_undegraded":"2024-03-16T20:11:54.996122+0000","last_fullsized":"2024-03-16T20:11:54.996122+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.995939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.995939+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.995939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:57.524502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"32'20","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.361012+0000","last_change":"2024-03-16T20:11:57.361084+0000","last_active":"2024-03-16T20:11:57.361012+0000","last_peered":"2024-03-16T20:11:57.361012+0000","last_clean":"2024-03-16T20:11:57.361012+0000","last_became_active":"2024-03-16T19:56:33.979979+0000","last_became_peered":"2024-03-16T19:56:33.979979+0000","last_unstale":"2024-03-16T20:11:57.361012+0000","last_undegraded":"2024-03-16T20:11:57.361012+0000","last_fullsized":"2024-03-16T20:11:57.361012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:57.360851+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:57.360851+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.360851+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:27.227797+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.189988+0000","last_change":"2024-03-16T20:11:57.190023+0000","last_active":"2024-03-16T20:11:57.189988+0000","last_peered":"2024-03-16T20:11:57.189988+0000","last_clean":"2024-03-16T20:11:57.189988+0000","last_became_active":"2024-03-16T19:56:36.304975+0000","last_became_peered":"2024-03-16T19:56:36.304975+0000","last_unstale":"2024-03-16T20:11:57.189988+0000","last_undegraded":"2024-03-16T20:11:57.189988+0000","last_fullsized":"2024-03-16T20:11:57.189988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.189894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.189894+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.189894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:40:18.144661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.959804+0000","last_change":"2024-03-16T20:11:55.959847+0000","last_active":"2024-03-16T20:11:55.959804+0000","last_peered":"2024-03-16T20:11:55.959804+0000","last_clean":"2024-03-16T20:11:55.959804+0000","last_became_active":"2024-03-16T19:56:43.465460+0000","last_became_peered":"2024-03-16T19:56:43.465460+0000","last_unstale":"2024-03-16T20:11:55.959804+0000","last_undegraded":"2024-03-16T20:11:55.959804+0000","last_fullsized":"2024-03-16T20:11:55.959804+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.959692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.959692+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.959692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:19.431538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.400540+0000","last_change":"2024-03-16T20:11:58.400593+0000","last_active":"2024-03-16T20:11:58.400540+0000","last_peered":"2024-03-16T20:11:58.400540+0000","last_clean":"2024-03-16T20:11:58.400540+0000","last_became_active":"2024-03-16T19:56:33.981554+0000","last_became_peered":"2024-03-16T19:56:33.981554+0000","last_unstale":"2024-03-16T20:11:58.400540+0000","last_undegraded":"2024-03-16T20:11:58.400540+0000","last_fullsized":"2024-03-16T20:11:58.400540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:58.400442+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:58.400442+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.400442+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:13.823169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.191318+0000","last_change":"2024-03-16T20:11:58.191355+0000","last_active":"2024-03-16T20:11:58.191318+0000","last_peered":"2024-03-16T20:11:58.191318+0000","last_clean":"2024-03-16T20:11:58.191318+0000","last_became_active":"2024-03-16T19:56:36.309154+0000","last_became_peered":"2024-03-16T19:56:36.309154+0000","last_unstale":"2024-03-16T20:11:58.191318+0000","last_undegraded":"2024-03-16T20:11:58.191318+0000","last_fullsized":"2024-03-16T20:11:58.191318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.191205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.191205+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.191205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:14:52.421910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.007217+0000","last_change":"2024-03-16T20:11:57.007260+0000","last_active":"2024-03-16T20:11:57.007217+0000","last_peered":"2024-03-16T20:11:57.007217+0000","last_clean":"2024-03-16T20:11:57.007217+0000","last_became_active":"2024-03-16T19:56:43.465616+0000","last_became_peered":"2024-03-16T19:56:43.465616+0000","last_unstale":"2024-03-16T20:11:57.007217+0000","last_undegraded":"2024-03-16T20:11:57.007217+0000","last_fullsized":"2024-03-16T20:11:57.007217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.007105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.007105+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.007105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:57.874060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.179505+0000","last_change":"2024-03-16T20:11:55.179632+0000","last_active":"2024-03-16T20:11:55.179505+0000","last_peered":"2024-03-16T20:11:55.179505+0000","last_clean":"2024-03-16T20:11:55.179505+0000","last_became_active":"2024-03-16T19:56:36.304422+0000","last_became_peered":"2024-03-16T19:56:36.304422+0000","last_unstale":"2024-03-16T20:11:55.179505+0000","last_undegraded":"2024-03-16T20:11:55.179505+0000","last_fullsized":"2024-03-16T20:11:55.179505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.179204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.179204+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.179204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:03:33.091999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"32'6","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.313926+0000","last_change":"2024-03-16T20:11:54.314000+0000","last_active":"2024-03-16T20:11:54.313926+0000","last_peered":"2024-03-16T20:11:54.313926+0000","last_clean":"2024-03-16T20:11:54.313926+0000","last_became_active":"2024-03-16T19:56:33.903635+0000","last_became_peered":"2024-03-16T19:56:33.903635+0000","last_unstale":"2024-03-16T20:11:54.313926+0000","last_undegraded":"2024-03-16T20:11:54.313926+0000","last_fullsized":"2024-03-16T20:11:54.313926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-03-16T20:11:54.313753+0000","last_deep_scrub":"32'6","last_deep_scrub_stamp":"2024-03-16T20:11:54.313753+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.313753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:17.785599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.941717+0000","last_change":"2024-03-16T20:11:52.941717+0000","last_active":"2024-03-16T20:11:52.941717+0000","last_peered":"2024-03-16T20:11:52.941717+0000","last_clean":"2024-03-16T20:11:52.941717+0000","last_became_active":"2024-03-16T19:56:43.436460+0000","last_became_peered":"2024-03-16T19:56:43.436460+0000","last_unstale":"2024-03-16T20:11:52.941717+0000","last_undegraded":"2024-03-16T20:11:52.941717+0000","last_fullsized":"2024-03-16T20:11:52.941717+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.941573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.941573+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.941573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:02:20.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.880238+0000","last_change":"2024-03-16T20:11:47.880238+0000","last_active":"2024-03-16T20:11:47.880238+0000","last_peered":"2024-03-16T20:11:47.880238+0000","last_clean":"2024-03-16T20:11:47.880238+0000","last_became_active":"2024-03-16T19:56:36.317802+0000","last_became_peered":"2024-03-16T19:56:36.317802+0000","last_unstale":"2024-03-16T20:11:47.880238+0000","last_undegraded":"2024-03-16T20:11:47.880238+0000","last_fullsized":"2024-03-16T20:11:47.880238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.880065+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.880065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:16.609260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"32'5","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.020055+0000","last_change":"2024-03-16T20:11:46.020079+0000","last_active":"2024-03-16T20:11:46.020055+0000","last_peered":"2024-03-16T20:11:46.020055+0000","last_clean":"2024-03-16T20:11:46.020055+0000","last_became_active":"2024-03-16T19:56:33.856932+0000","last_became_peered":"2024-03-16T19:56:33.856932+0000","last_unstale":"2024-03-16T20:11:46.020055+0000","last_undegraded":"2024-03-16T20:11:46.020055+0000","last_fullsized":"2024-03-16T20:11:46.020055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:46.019995+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:46.019995+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.019995+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:08:53.109964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.988704+0000","last_change":"2024-03-16T20:11:57.988704+0000","last_active":"2024-03-16T20:11:57.988704+0000","last_peered":"2024-03-16T20:11:57.988704+0000","last_clean":"2024-03-16T20:11:57.988704+0000","last_became_active":"2024-03-16T19:56:43.465379+0000","last_became_peered":"2024-03-16T19:56:43.465379+0000","last_unstale":"2024-03-16T20:11:57.988704+0000","last_undegraded":"2024-03-16T20:11:57.988704+0000","last_fullsized":"2024-03-16T20:11:57.988704+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.988554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.988554+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.988554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:15.925804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.918364+0000","last_change":"2024-03-16T20:11:52.918397+0000","last_active":"2024-03-16T20:11:52.918364+0000","last_peered":"2024-03-16T20:11:52.918364+0000","last_clean":"2024-03-16T20:11:52.918364+0000","last_became_active":"2024-03-16T19:56:36.221223+0000","last_became_peered":"2024-03-16T19:56:36.221223+0000","last_unstale":"2024-03-16T20:11:52.918364+0000","last_undegraded":"2024-03-16T20:11:52.918364+0000","last_fullsized":"2024-03-16T20:11:52.918364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.918271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.918271+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.918271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:01.131831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.066808+0000","last_change":"2024-03-16T20:11:47.066963+0000","last_active":"2024-03-16T20:11:47.066808+0000","last_peered":"2024-03-16T20:11:47.066808+0000","last_clean":"2024-03-16T20:11:47.066808+0000","last_became_active":"2024-03-16T19:56:33.904438+0000","last_became_peered":"2024-03-16T19:56:33.904438+0000","last_unstale":"2024-03-16T20:11:47.066808+0000","last_undegraded":"2024-03-16T20:11:47.066808+0000","last_fullsized":"2024-03-16T20:11:47.066808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:47.066603+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:47.066603+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.066603+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:51:21.887232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.915147+0000","last_change":"2024-03-16T20:11:59.915147+0000","last_active":"2024-03-16T20:11:59.915147+0000","last_peered":"2024-03-16T20:11:59.915147+0000","last_clean":"2024-03-16T20:11:59.915147+0000","last_became_active":"2024-03-16T19:56:43.496108+0000","last_became_peered":"2024-03-16T19:56:43.496108+0000","last_unstale":"2024-03-16T20:11:59.915147+0000","last_undegraded":"2024-03-16T20:11:59.915147+0000","last_fullsized":"2024-03-16T20:11:59.915147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.914883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.914883+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.914883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:12.338529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.003474+0000","last_change":"2024-03-16T20:11:50.003514+0000","last_active":"2024-03-16T20:11:50.003474+0000","last_peered":"2024-03-16T20:11:50.003474+0000","last_clean":"2024-03-16T20:11:50.003474+0000","last_became_active":"2024-03-16T19:56:36.249643+0000","last_became_peered":"2024-03-16T19:56:36.249643+0000","last_unstale":"2024-03-16T20:11:50.003474+0000","last_undegraded":"2024-03-16T20:11:50.003474+0000","last_fullsized":"2024-03-16T20:11:50.003474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.003363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.003363+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.003363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:12:20.749265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"32'3","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.357139+0000","last_change":"2024-03-16T20:11:55.357185+0000","last_active":"2024-03-16T20:11:55.357139+0000","last_peered":"2024-03-16T20:11:55.357139+0000","last_clean":"2024-03-16T20:11:55.357139+0000","last_became_active":"2024-03-16T19:56:33.966452+0000","last_became_peered":"2024-03-16T19:56:33.966452+0000","last_unstale":"2024-03-16T20:11:55.357139+0000","last_undegraded":"2024-03-16T20:11:55.357139+0000","last_fullsized":"2024-03-16T20:11:55.357139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3","last_scrub_stamp":"2024-03-16T20:11:55.357043+0000","last_deep_scrub":"32'3","last_deep_scrub_stamp":"2024-03-16T20:11:55.357043+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.357043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:56.223998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.961790+0000","last_change":"2024-03-16T20:11:58.961790+0000","last_active":"2024-03-16T20:11:58.961790+0000","last_peered":"2024-03-16T20:11:58.961790+0000","last_clean":"2024-03-16T20:11:58.961790+0000","last_became_active":"2024-03-16T19:56:43.467248+0000","last_became_peered":"2024-03-16T19:56:43.467248+0000","last_unstale":"2024-03-16T20:11:58.961790+0000","last_undegraded":"2024-03-16T20:11:58.961790+0000","last_fullsized":"2024-03-16T20:11:58.961790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.961630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.961630+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.961630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:28.828577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.797979+0000","last_change":"2024-03-16T20:11:48.798017+0000","last_active":"2024-03-16T20:11:48.797979+0000","last_peered":"2024-03-16T20:11:48.797979+0000","last_clean":"2024-03-16T20:11:48.797979+0000","last_became_active":"2024-03-16T19:56:36.312528+0000","last_became_peered":"2024-03-16T19:56:36.312528+0000","last_unstale":"2024-03-16T20:11:48.797979+0000","last_undegraded":"2024-03-16T20:11:48.797979+0000","last_fullsized":"2024-03-16T20:11:48.797979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.797866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.797866+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.797866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:12.269712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.185301+0000","last_change":"2024-03-16T20:11:50.185372+0000","last_active":"2024-03-16T20:11:50.185301+0000","last_peered":"2024-03-16T20:11:50.185301+0000","last_clean":"2024-03-16T20:11:50.185301+0000","last_became_active":"2024-03-16T19:56:33.971227+0000","last_became_peered":"2024-03-16T19:56:33.971227+0000","last_unstale":"2024-03-16T20:11:50.185301+0000","last_undegraded":"2024-03-16T20:11:50.185301+0000","last_fullsized":"2024-03-16T20:11:50.185301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:50.185142+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:50.185142+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.185142+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:27.757147+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.955818+0000","last_change":"2024-03-16T20:11:59.955860+0000","last_active":"2024-03-16T20:11:59.955818+0000","last_peered":"2024-03-16T20:11:59.955818+0000","last_clean":"2024-03-16T20:11:59.955818+0000","last_became_active":"2024-03-16T19:56:43.489229+0000","last_became_peered":"2024-03-16T19:56:43.489229+0000","last_unstale":"2024-03-16T20:11:59.955818+0000","last_undegraded":"2024-03-16T20:11:59.955818+0000","last_fullsized":"2024-03-16T20:11:59.955818+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.955695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.955695+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.955695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:15.378423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.800890+0000","last_change":"2024-03-16T20:11:53.800929+0000","last_active":"2024-03-16T20:11:53.800890+0000","last_peered":"2024-03-16T20:11:53.800890+0000","last_clean":"2024-03-16T20:11:53.800890+0000","last_became_active":"2024-03-16T19:57:41.287118+0000","last_became_peered":"2024-03-16T19:57:41.287118+0000","last_unstale":"2024-03-16T20:11:53.800890+0000","last_undegraded":"2024-03-16T20:11:53.800890+0000","last_fullsized":"2024-03-16T20:11:53.800890+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.800778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.800778+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.800778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:00.496635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.134129+0000","last_change":"2024-03-16T20:11:49.134180+0000","last_active":"2024-03-16T20:11:49.134129+0000","last_peered":"2024-03-16T20:11:49.134129+0000","last_clean":"2024-03-16T20:11:49.134129+0000","last_became_active":"2024-03-16T19:56:33.975003+0000","last_became_peered":"2024-03-16T19:56:33.975003+0000","last_unstale":"2024-03-16T20:11:49.134129+0000","last_undegraded":"2024-03-16T20:11:49.134129+0000","last_fullsized":"2024-03-16T20:11:49.134129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:49.134032+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:49.134032+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.134032+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:05.343809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.825013+0000","last_change":"2024-03-16T20:12:00.825054+0000","last_active":"2024-03-16T20:12:00.825013+0000","last_peered":"2024-03-16T20:12:00.825013+0000","last_clean":"2024-03-16T20:12:00.825013+0000","last_became_active":"2024-03-16T19:56:43.491490+0000","last_became_peered":"2024-03-16T19:56:43.491490+0000","last_unstale":"2024-03-16T20:12:00.825013+0000","last_undegraded":"2024-03-16T20:12:00.825013+0000","last_fullsized":"2024-03-16T20:12:00.825013+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.824902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.824902+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.824902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:37.127249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.948757+0000","last_change":"2024-03-16T20:11:43.948796+0000","last_active":"2024-03-16T20:11:43.948757+0000","last_peered":"2024-03-16T20:11:43.948757+0000","last_clean":"2024-03-16T20:11:43.948757+0000","last_became_active":"2024-03-16T19:56:36.234407+0000","last_became_peered":"2024-03-16T19:56:36.234407+0000","last_unstale":"2024-03-16T20:11:43.948757+0000","last_undegraded":"2024-03-16T20:11:43.948757+0000","last_fullsized":"2024-03-16T20:11:43.948757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:43.948646+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.948646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:36:25.189231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"32'7","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.344682+0000","last_change":"2024-03-16T20:11:56.344733+0000","last_active":"2024-03-16T20:11:56.344682+0000","last_peered":"2024-03-16T20:11:56.344682+0000","last_clean":"2024-03-16T20:11:56.344682+0000","last_became_active":"2024-03-16T19:56:33.967175+0000","last_became_peered":"2024-03-16T19:56:33.967175+0000","last_unstale":"2024-03-16T20:11:56.344682+0000","last_undegraded":"2024-03-16T20:11:56.344682+0000","last_fullsized":"2024-03-16T20:11:56.344682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:56.344588+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:56.344588+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.344588+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:14.281837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.828724+0000","last_change":"2024-03-16T20:11:55.828765+0000","last_active":"2024-03-16T20:11:55.828724+0000","last_peered":"2024-03-16T20:11:55.828724+0000","last_clean":"2024-03-16T20:11:55.828724+0000","last_became_active":"2024-03-16T19:56:43.486393+0000","last_became_peered":"2024-03-16T19:56:43.486393+0000","last_unstale":"2024-03-16T20:11:55.828724+0000","last_undegraded":"2024-03-16T20:11:55.828724+0000","last_fullsized":"2024-03-16T20:11:55.828724+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.828618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.828618+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.828618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:42.946999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.285214+0000","last_change":"2024-03-16T20:11:48.285251+0000","last_active":"2024-03-16T20:11:48.285214+0000","last_peered":"2024-03-16T20:11:48.285214+0000","last_clean":"2024-03-16T20:11:48.285214+0000","last_became_active":"2024-03-16T19:56:36.306521+0000","last_became_peered":"2024-03-16T19:56:36.306521+0000","last_unstale":"2024-03-16T20:11:48.285214+0000","last_undegraded":"2024-03-16T20:11:48.285214+0000","last_fullsized":"2024-03-16T20:11:48.285214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.285097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.285097+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.285097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:09.534808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":52,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.089469+0000","last_change":"2024-03-16T20:11:48.089513+0000","last_active":"2024-03-16T20:11:48.089469+0000","last_peered":"2024-03-16T20:11:48.089469+0000","last_clean":"2024-03-16T20:11:48.089469+0000","last_became_active":"2024-03-16T19:56:33.964105+0000","last_became_peered":"2024-03-16T19:56:33.964105+0000","last_unstale":"2024-03-16T20:11:48.089469+0000","last_undegraded":"2024-03-16T20:11:48.089469+0000","last_fullsized":"2024-03-16T20:11:48.089469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'1","last_scrub_stamp":"2024-03-16T20:11:48.089355+0000","last_deep_scrub":"29'1","last_deep_scrub_stamp":"2024-03-16T20:11:48.089355+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.089355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:22:35.616024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.977722+0000","last_change":"2024-03-16T20:11:56.977766+0000","last_active":"2024-03-16T20:11:56.977722+0000","last_peered":"2024-03-16T20:11:56.977722+0000","last_clean":"2024-03-16T20:11:56.977722+0000","last_became_active":"2024-03-16T19:56:43.456190+0000","last_became_peered":"2024-03-16T19:56:43.456190+0000","last_unstale":"2024-03-16T20:11:56.977722+0000","last_undegraded":"2024-03-16T20:11:56.977722+0000","last_fullsized":"2024-03-16T20:11:56.977722+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.977595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.977595+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.977595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:58:33.889972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.985107+0000","last_change":"2024-03-16T20:11:50.985145+0000","last_active":"2024-03-16T20:11:50.985107+0000","last_peered":"2024-03-16T20:11:50.985107+0000","last_clean":"2024-03-16T20:11:50.985107+0000","last_became_active":"2024-03-16T19:56:36.302131+0000","last_became_peered":"2024-03-16T19:56:36.302131+0000","last_unstale":"2024-03-16T20:11:50.985107+0000","last_undegraded":"2024-03-16T20:11:50.985107+0000","last_fullsized":"2024-03-16T20:11:50.985107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.985004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.985004+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.985004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:31.441667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"32'8","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.207803+0000","last_change":"2024-03-16T20:11:52.207870+0000","last_active":"2024-03-16T20:11:52.207803+0000","last_peered":"2024-03-16T20:11:52.207803+0000","last_clean":"2024-03-16T20:11:52.207803+0000","last_became_active":"2024-03-16T19:56:33.975305+0000","last_became_peered":"2024-03-16T19:56:33.975305+0000","last_unstale":"2024-03-16T20:11:52.207803+0000","last_undegraded":"2024-03-16T20:11:52.207803+0000","last_fullsized":"2024-03-16T20:11:52.207803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:52.207634+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:52.207634+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.207634+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:51:47.529032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.963647+0000","last_change":"2024-03-16T20:11:57.963689+0000","last_active":"2024-03-16T20:11:57.963647+0000","last_peered":"2024-03-16T20:11:57.963647+0000","last_clean":"2024-03-16T20:11:57.963647+0000","last_became_active":"2024-03-16T19:56:43.459914+0000","last_became_peered":"2024-03-16T19:56:43.459914+0000","last_unstale":"2024-03-16T20:11:57.963647+0000","last_undegraded":"2024-03-16T20:11:57.963647+0000","last_fullsized":"2024-03-16T20:11:57.963647+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.963536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.963536+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.963536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:03:09.515499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.011073+0000","last_change":"2024-03-16T20:11:53.011109+0000","last_active":"2024-03-16T20:11:53.011073+0000","last_peered":"2024-03-16T20:11:53.011073+0000","last_clean":"2024-03-16T20:11:53.011073+0000","last_became_active":"2024-03-16T19:56:36.234406+0000","last_became_peered":"2024-03-16T19:56:36.234406+0000","last_unstale":"2024-03-16T20:11:53.011073+0000","last_undegraded":"2024-03-16T20:11:53.011073+0000","last_fullsized":"2024-03-16T20:11:53.011073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.010964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.010964+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.010964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:19.139131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"32'8","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:46.876745+0000","last_change":"2024-03-16T20:11:46.876796+0000","last_active":"2024-03-16T20:11:46.876745+0000","last_peered":"2024-03-16T20:11:46.876745+0000","last_clean":"2024-03-16T20:11:46.876745+0000","last_became_active":"2024-03-16T19:56:33.900107+0000","last_became_peered":"2024-03-16T19:56:33.900107+0000","last_unstale":"2024-03-16T20:11:46.876745+0000","last_undegraded":"2024-03-16T20:11:46.876745+0000","last_fullsized":"2024-03-16T20:11:46.876745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:46.876643+0000","last_clean_scrub_stamp":"2024-03-16T20:11:46.876643+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:35.335725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.925571+0000","last_change":"2024-03-16T20:11:58.925719+0000","last_active":"2024-03-16T20:11:58.925571+0000","last_peered":"2024-03-16T20:11:58.925571+0000","last_clean":"2024-03-16T20:11:58.925571+0000","last_became_active":"2024-03-16T19:56:43.466720+0000","last_became_peered":"2024-03-16T19:56:43.466720+0000","last_unstale":"2024-03-16T20:11:58.925571+0000","last_undegraded":"2024-03-16T20:11:58.925571+0000","last_fullsized":"2024-03-16T20:11:58.925571+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.925363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.925363+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.925363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:39.387673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.875844+0000","last_change":"2024-03-16T20:11:50.875881+0000","last_active":"2024-03-16T20:11:50.875844+0000","last_peered":"2024-03-16T20:11:50.875844+0000","last_clean":"2024-03-16T20:11:50.875844+0000","last_became_active":"2024-03-16T19:56:36.313128+0000","last_became_peered":"2024-03-16T19:56:36.313128+0000","last_unstale":"2024-03-16T20:11:50.875844+0000","last_undegraded":"2024-03-16T20:11:50.875844+0000","last_fullsized":"2024-03-16T20:11:50.875844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.875738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.875738+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.875738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:08.812988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"32'5","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.302588+0000","last_change":"2024-03-16T20:11:53.302635+0000","last_active":"2024-03-16T20:11:53.302588+0000","last_peered":"2024-03-16T20:11:53.302588+0000","last_clean":"2024-03-16T20:11:53.302588+0000","last_became_active":"2024-03-16T19:56:33.902108+0000","last_became_peered":"2024-03-16T19:56:33.902108+0000","last_unstale":"2024-03-16T20:11:53.302588+0000","last_undegraded":"2024-03-16T20:11:53.302588+0000","last_fullsized":"2024-03-16T20:11:53.302588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:53.302493+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:53.302493+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.302493+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:55.831055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.908287+0000","last_change":"2024-03-16T20:11:57.908329+0000","last_active":"2024-03-16T20:11:57.908287+0000","last_peered":"2024-03-16T20:11:57.908287+0000","last_clean":"2024-03-16T20:11:57.908287+0000","last_became_active":"2024-03-16T19:56:43.494675+0000","last_became_peered":"2024-03-16T19:56:43.494675+0000","last_unstale":"2024-03-16T20:11:57.908287+0000","last_undegraded":"2024-03-16T20:11:57.908287+0000","last_fullsized":"2024-03-16T20:11:57.908287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.908177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.908177+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.908177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:49:28.348900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.898622+0000","last_change":"2024-03-16T20:12:03.898661+0000","last_active":"2024-03-16T20:12:03.898622+0000","last_peered":"2024-03-16T20:12:03.898622+0000","last_clean":"2024-03-16T20:12:03.898622+0000","last_became_active":"2024-03-16T19:57:41.238526+0000","last_became_peered":"2024-03-16T19:57:41.238526+0000","last_unstale":"2024-03-16T20:12:03.898622+0000","last_undegraded":"2024-03-16T20:12:03.898622+0000","last_fullsized":"2024-03-16T20:12:03.898622+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.898514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.898514+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.898514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:55.718023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"32'11","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.241988+0000","last_change":"2024-03-16T20:11:51.242042+0000","last_active":"2024-03-16T20:11:51.241988+0000","last_peered":"2024-03-16T20:11:51.241988+0000","last_clean":"2024-03-16T20:11:51.241988+0000","last_became_active":"2024-03-16T19:56:33.970993+0000","last_became_peered":"2024-03-16T19:56:33.970993+0000","last_unstale":"2024-03-16T20:11:51.241988+0000","last_undegraded":"2024-03-16T20:11:51.241988+0000","last_fullsized":"2024-03-16T20:11:51.241988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'11","last_scrub_stamp":"2024-03-16T20:11:51.241901+0000","last_deep_scrub":"32'11","last_deep_scrub_stamp":"2024-03-16T20:11:51.241901+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.241901+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:53.770463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'119","reported_seq":209,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.276981+0000","last_change":"2024-03-16T20:11:52.276981+0000","last_active":"2024-03-16T20:11:52.276981+0000","last_peered":"2024-03-16T20:11:52.276981+0000","last_clean":"2024-03-16T20:11:52.276981+0000","last_became_active":"2024-03-16T19:55:55.092298+0000","last_became_peered":"2024-03-16T19:55:55.092298+0000","last_unstale":"2024-03-16T20:11:52.276981+0000","last_undegraded":"2024-03-16T20:11:52.276981+0000","last_fullsized":"2024-03-16T20:11:52.276981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'119","last_scrub_stamp":"2024-03-16T20:11:52.276821+0000","last_deep_scrub":"32'119","last_deep_scrub_stamp":"2024-03-16T20:11:52.276821+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.276821+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:58.972679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.191201+0000","last_change":"2024-03-16T20:12:02.191243+0000","last_active":"2024-03-16T20:12:02.191201+0000","last_peered":"2024-03-16T20:12:02.191201+0000","last_clean":"2024-03-16T20:12:02.191201+0000","last_became_active":"2024-03-16T19:56:43.486315+0000","last_became_peered":"2024-03-16T19:56:43.486315+0000","last_unstale":"2024-03-16T20:12:02.191201+0000","last_undegraded":"2024-03-16T20:12:02.191201+0000","last_fullsized":"2024-03-16T20:12:02.191201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.191077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.191077+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.191077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:49.309458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:05.459874+0000","last_change":"2024-03-16T20:12:05.459874+0000","last_active":"2024-03-16T20:12:05.459874+0000","last_peered":"2024-03-16T20:12:05.459874+0000","last_clean":"2024-03-16T20:12:05.459874+0000","last_became_active":"2024-03-16T19:56:36.298389+0000","last_became_peered":"2024-03-16T19:56:36.298389+0000","last_unstale":"2024-03-16T20:12:05.459874+0000","last_undegraded":"2024-03-16T20:12:05.459874+0000","last_fullsized":"2024-03-16T20:12:05.459874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:05.459685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:05.459685+0000","last_clean_scrub_stamp":"2024-03-16T20:12:05.459685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:49:06.206686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"32'8","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.907475+0000","last_change":"2024-03-16T20:11:47.907475+0000","last_active":"2024-03-16T20:11:47.907475+0000","last_peered":"2024-03-16T20:11:47.907475+0000","last_clean":"2024-03-16T20:11:47.907475+0000","last_became_active":"2024-03-16T19:56:33.984448+0000","last_became_peered":"2024-03-16T19:56:33.984448+0000","last_unstale":"2024-03-16T20:11:47.907475+0000","last_undegraded":"2024-03-16T20:11:47.907475+0000","last_fullsized":"2024-03-16T20:11:47.907475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:47.907316+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.907316+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:21:14.912976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.177252+0000","last_change":"2024-03-16T20:12:01.177292+0000","last_active":"2024-03-16T20:12:01.177252+0000","last_peered":"2024-03-16T20:12:01.177252+0000","last_clean":"2024-03-16T20:12:01.177252+0000","last_became_active":"2024-03-16T19:56:43.483922+0000","last_became_peered":"2024-03-16T19:56:43.483922+0000","last_unstale":"2024-03-16T20:12:01.177252+0000","last_undegraded":"2024-03-16T20:12:01.177252+0000","last_fullsized":"2024-03-16T20:12:01.177252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.177140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.177140+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.177140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:09.390700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.186426+0000","last_change":"2024-03-16T20:11:56.186426+0000","last_active":"2024-03-16T20:11:56.186426+0000","last_peered":"2024-03-16T20:11:56.186426+0000","last_clean":"2024-03-16T20:11:56.186426+0000","last_became_active":"2024-03-16T19:56:36.299677+0000","last_became_peered":"2024-03-16T19:56:36.299677+0000","last_unstale":"2024-03-16T20:11:56.186426+0000","last_undegraded":"2024-03-16T20:11:56.186426+0000","last_fullsized":"2024-03-16T20:11:56.186426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.186272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.186272+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.186272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:47.836647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"32'4","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.993895+0000","last_change":"2024-03-16T20:11:44.993895+0000","last_active":"2024-03-16T20:11:44.993895+0000","last_peered":"2024-03-16T20:11:44.993895+0000","last_clean":"2024-03-16T20:11:44.993895+0000","last_became_active":"2024-03-16T19:56:33.856411+0000","last_became_peered":"2024-03-16T19:56:33.856411+0000","last_unstale":"2024-03-16T20:11:44.993895+0000","last_undegraded":"2024-03-16T20:11:44.993895+0000","last_fullsized":"2024-03-16T20:11:44.993895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4","last_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_deep_scrub":"32'4","last_deep_scrub_stamp":"2024-03-16T20:11:44.993741+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.993741+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:31:57.200016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.885962+0000","last_change":"2024-03-16T20:11:53.885962+0000","last_active":"2024-03-16T20:11:53.885962+0000","last_peered":"2024-03-16T20:11:53.885962+0000","last_clean":"2024-03-16T20:11:53.885962+0000","last_became_active":"2024-03-16T19:56:43.466480+0000","last_became_peered":"2024-03-16T19:56:43.466480+0000","last_unstale":"2024-03-16T20:11:53.885962+0000","last_undegraded":"2024-03-16T20:11:53.885962+0000","last_fullsized":"2024-03-16T20:11:53.885962+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.885809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.885809+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.885809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:08.690092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.418462+0000","last_change":"2024-03-16T20:12:02.418462+0000","last_active":"2024-03-16T20:12:02.418462+0000","last_peered":"2024-03-16T20:12:02.418462+0000","last_clean":"2024-03-16T20:12:02.418462+0000","last_became_active":"2024-03-16T19:56:36.253599+0000","last_became_peered":"2024-03-16T19:56:36.253599+0000","last_unstale":"2024-03-16T20:12:02.418462+0000","last_undegraded":"2024-03-16T20:12:02.418462+0000","last_fullsized":"2024-03-16T20:12:02.418462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.418214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.418214+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.418214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:27.394802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"32'7","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.836458+0000","last_change":"2024-03-16T20:11:44.836458+0000","last_active":"2024-03-16T20:11:44.836458+0000","last_peered":"2024-03-16T20:11:44.836458+0000","last_clean":"2024-03-16T20:11:44.836458+0000","last_became_active":"2024-03-16T19:56:33.899600+0000","last_became_peered":"2024-03-16T19:56:33.899600+0000","last_unstale":"2024-03-16T20:11:44.836458+0000","last_undegraded":"2024-03-16T20:11:44.836458+0000","last_fullsized":"2024-03-16T20:11:44.836458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:44.836292+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.836292+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:01.699700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:05.222794+0000","last_change":"2024-03-16T20:12:05.222840+0000","last_active":"2024-03-16T20:12:05.222794+0000","last_peered":"2024-03-16T20:12:05.222794+0000","last_clean":"2024-03-16T20:12:05.222794+0000","last_became_active":"2024-03-16T19:56:43.491766+0000","last_became_peered":"2024-03-16T19:56:43.491766+0000","last_unstale":"2024-03-16T20:12:05.222794+0000","last_undegraded":"2024-03-16T20:12:05.222794+0000","last_fullsized":"2024-03-16T20:12:05.222794+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:05.222663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:05.222663+0000","last_clean_scrub_stamp":"2024-03-16T20:12:05.222663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:41.525010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.418170+0000","last_change":"2024-03-16T20:12:01.418170+0000","last_active":"2024-03-16T20:12:01.418170+0000","last_peered":"2024-03-16T20:12:01.418170+0000","last_clean":"2024-03-16T20:12:01.418170+0000","last_became_active":"2024-03-16T19:56:36.250328+0000","last_became_peered":"2024-03-16T19:56:36.250328+0000","last_unstale":"2024-03-16T20:12:01.418170+0000","last_undegraded":"2024-03-16T20:12:01.418170+0000","last_fullsized":"2024-03-16T20:12:01.418170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.417940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.417940+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.417940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:48.536321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"32'2","reported_seq":52,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.048781+0000","last_change":"2024-03-16T20:11:47.048781+0000","last_active":"2024-03-16T20:11:47.048781+0000","last_peered":"2024-03-16T20:11:47.048781+0000","last_clean":"2024-03-16T20:11:47.048781+0000","last_became_active":"2024-03-16T19:56:33.960485+0000","last_became_peered":"2024-03-16T19:56:33.960485+0000","last_unstale":"2024-03-16T20:11:47.048781+0000","last_undegraded":"2024-03-16T20:11:47.048781+0000","last_fullsized":"2024-03-16T20:11:47.048781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-03-16T20:11:47.048629+0000","last_deep_scrub":"32'2","last_deep_scrub_stamp":"2024-03-16T20:11:47.048629+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.048629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:35.507712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.949114+0000","last_change":"2024-03-16T20:12:00.949155+0000","last_active":"2024-03-16T20:12:00.949114+0000","last_peered":"2024-03-16T20:12:00.949114+0000","last_clean":"2024-03-16T20:12:00.949114+0000","last_became_active":"2024-03-16T19:57:41.242517+0000","last_became_peered":"2024-03-16T19:57:41.242517+0000","last_unstale":"2024-03-16T20:12:00.949114+0000","last_undegraded":"2024-03-16T20:12:00.949114+0000","last_fullsized":"2024-03-16T20:12:00.949114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.948999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.948999+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.948999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:15.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.431977+0000","last_change":"2024-03-16T20:12:03.432016+0000","last_active":"2024-03-16T20:12:03.431977+0000","last_peered":"2024-03-16T20:12:03.431977+0000","last_clean":"2024-03-16T20:12:03.431977+0000","last_became_active":"2024-03-16T19:56:36.256036+0000","last_became_peered":"2024-03-16T19:56:36.256036+0000","last_unstale":"2024-03-16T20:12:03.431977+0000","last_undegraded":"2024-03-16T20:12:03.431977+0000","last_fullsized":"2024-03-16T20:12:03.431977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.431867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.431867+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.431867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:29.674096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"32'26","reported_seq":77,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.873458+0000","last_change":"2024-03-16T20:11:45.873518+0000","last_active":"2024-03-16T20:11:45.873458+0000","last_peered":"2024-03-16T20:11:45.873458+0000","last_clean":"2024-03-16T20:11:45.873458+0000","last_became_active":"2024-03-16T19:56:33.900050+0000","last_became_peered":"2024-03-16T19:56:33.900050+0000","last_unstale":"2024-03-16T20:11:45.873458+0000","last_undegraded":"2024-03-16T20:11:45.873458+0000","last_fullsized":"2024-03-16T20:11:45.873458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'26","last_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_deep_scrub":"32'26","last_deep_scrub_stamp":"2024-03-16T20:11:45.873355+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.873355+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:46.491674+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.855277+0000","last_change":"2024-03-16T20:11:54.855338+0000","last_active":"2024-03-16T20:11:54.855277+0000","last_peered":"2024-03-16T20:11:54.855277+0000","last_clean":"2024-03-16T20:11:54.855277+0000","last_became_active":"2024-03-16T19:56:43.466333+0000","last_became_peered":"2024-03-16T19:56:43.466333+0000","last_unstale":"2024-03-16T20:11:54.855277+0000","last_undegraded":"2024-03-16T20:11:54.855277+0000","last_fullsized":"2024-03-16T20:11:54.855277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.855076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.855076+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.855076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:03.068636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.218998+0000","last_change":"2024-03-16T20:11:58.219034+0000","last_active":"2024-03-16T20:11:58.218998+0000","last_peered":"2024-03-16T20:11:58.218998+0000","last_clean":"2024-03-16T20:11:58.218998+0000","last_became_active":"2024-03-16T19:56:36.316140+0000","last_became_peered":"2024-03-16T19:56:36.316140+0000","last_unstale":"2024-03-16T20:11:58.218998+0000","last_undegraded":"2024-03-16T20:11:58.218998+0000","last_fullsized":"2024-03-16T20:11:58.218998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.218885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.218885+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.218885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:28.085832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"32'9","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.170572+0000","last_change":"2024-03-16T20:11:52.170620+0000","last_active":"2024-03-16T20:11:52.170572+0000","last_peered":"2024-03-16T20:11:52.170572+0000","last_clean":"2024-03-16T20:11:52.170572+0000","last_became_active":"2024-03-16T19:56:33.975219+0000","last_became_peered":"2024-03-16T19:56:33.975219+0000","last_unstale":"2024-03-16T20:11:52.170572+0000","last_undegraded":"2024-03-16T20:11:52.170572+0000","last_fullsized":"2024-03-16T20:11:52.170572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'9","last_scrub_stamp":"2024-03-16T20:11:52.170475+0000","last_deep_scrub":"32'9","last_deep_scrub_stamp":"2024-03-16T20:11:52.170475+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.170475+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:18.620366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:58.856029+0000","last_change":"2024-03-16T20:11:58.856070+0000","last_active":"2024-03-16T20:11:58.856029+0000","last_peered":"2024-03-16T20:11:58.856029+0000","last_clean":"2024-03-16T20:11:58.856029+0000","last_became_active":"2024-03-16T19:56:43.489413+0000","last_became_peered":"2024-03-16T19:56:43.489413+0000","last_unstale":"2024-03-16T20:11:58.856029+0000","last_undegraded":"2024-03-16T20:11:58.856029+0000","last_fullsized":"2024-03-16T20:11:58.856029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:58.855920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:58.855920+0000","last_clean_scrub_stamp":"2024-03-16T20:11:58.855920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:08:03.774971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.382315+0000","last_change":"2024-03-16T20:12:00.382369+0000","last_active":"2024-03-16T20:12:00.382315+0000","last_peered":"2024-03-16T20:12:00.382315+0000","last_clean":"2024-03-16T20:12:00.382315+0000","last_became_active":"2024-03-16T19:56:36.249884+0000","last_became_peered":"2024-03-16T19:56:36.249884+0000","last_unstale":"2024-03-16T20:12:00.382315+0000","last_undegraded":"2024-03-16T20:12:00.382315+0000","last_fullsized":"2024-03-16T20:12:00.382315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.382140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.382140+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.382140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:08.595700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"32'8","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.271165+0000","last_change":"2024-03-16T20:11:50.271224+0000","last_active":"2024-03-16T20:11:50.271165+0000","last_peered":"2024-03-16T20:11:50.271165+0000","last_clean":"2024-03-16T20:11:50.271165+0000","last_became_active":"2024-03-16T19:56:33.970424+0000","last_became_peered":"2024-03-16T19:56:33.970424+0000","last_unstale":"2024-03-16T20:11:50.271165+0000","last_undegraded":"2024-03-16T20:11:50.271165+0000","last_fullsized":"2024-03-16T20:11:50.271165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'8","last_scrub_stamp":"2024-03-16T20:11:50.271033+0000","last_deep_scrub":"32'8","last_deep_scrub_stamp":"2024-03-16T20:11:50.271033+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.271033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:56.966891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.854296+0000","last_change":"2024-03-16T20:11:59.854336+0000","last_active":"2024-03-16T20:11:59.854296+0000","last_peered":"2024-03-16T20:11:59.854296+0000","last_clean":"2024-03-16T20:11:59.854296+0000","last_became_active":"2024-03-16T19:56:43.491157+0000","last_became_peered":"2024-03-16T19:56:43.491157+0000","last_unstale":"2024-03-16T20:11:59.854296+0000","last_undegraded":"2024-03-16T20:11:59.854296+0000","last_fullsized":"2024-03-16T20:11:59.854296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.854176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.854176+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.854176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:12:13.481841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.842717+0000","last_change":"2024-03-16T20:11:52.842769+0000","last_active":"2024-03-16T20:11:52.842717+0000","last_peered":"2024-03-16T20:11:52.842717+0000","last_clean":"2024-03-16T20:11:52.842717+0000","last_became_active":"2024-03-16T19:57:41.286966+0000","last_became_peered":"2024-03-16T19:57:41.286966+0000","last_unstale":"2024-03-16T20:11:52.842717+0000","last_undegraded":"2024-03-16T20:11:52.842717+0000","last_fullsized":"2024-03-16T20:11:52.842717+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.842542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.842542+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.842542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:12.552194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"32'15","reported_seq":65,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:43.686996+0000","last_change":"2024-03-16T20:11:43.687053+0000","last_active":"2024-03-16T20:11:43.686996+0000","last_peered":"2024-03-16T20:11:43.686996+0000","last_clean":"2024-03-16T20:11:43.686996+0000","last_became_active":"2024-03-16T19:56:33.972714+0000","last_became_peered":"2024-03-16T19:56:33.972714+0000","last_unstale":"2024-03-16T20:11:43.686996+0000","last_undegraded":"2024-03-16T20:11:43.686996+0000","last_fullsized":"2024-03-16T20:11:43.686996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'15","last_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_deep_scrub":"32'15","last_deep_scrub_stamp":"2024-03-16T20:11:43.686892+0000","last_clean_scrub_stamp":"2024-03-16T20:11:43.686892+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:16.356150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:04.217553+0000","last_change":"2024-03-16T20:12:04.217601+0000","last_active":"2024-03-16T20:12:04.217553+0000","last_peered":"2024-03-16T20:12:04.217553+0000","last_clean":"2024-03-16T20:12:04.217553+0000","last_became_active":"2024-03-16T19:56:43.489120+0000","last_became_peered":"2024-03-16T19:56:43.489120+0000","last_unstale":"2024-03-16T20:12:04.217553+0000","last_undegraded":"2024-03-16T20:12:04.217553+0000","last_fullsized":"2024-03-16T20:12:04.217553+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:04.217414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:04.217414+0000","last_clean_scrub_stamp":"2024-03-16T20:12:04.217414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:12:17.845079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.402127+0000","last_change":"2024-03-16T20:11:59.402179+0000","last_active":"2024-03-16T20:11:59.402127+0000","last_peered":"2024-03-16T20:11:59.402127+0000","last_clean":"2024-03-16T20:11:59.402127+0000","last_became_active":"2024-03-16T19:56:36.213080+0000","last_became_peered":"2024-03-16T19:56:36.213080+0000","last_unstale":"2024-03-16T20:11:59.402127+0000","last_undegraded":"2024-03-16T20:11:59.402127+0000","last_fullsized":"2024-03-16T20:11:59.402127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.401946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.401946+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.401946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:44:14.206230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"32'20","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.222202+0000","last_change":"2024-03-16T20:11:49.222234+0000","last_active":"2024-03-16T20:11:49.222202+0000","last_peered":"2024-03-16T20:11:49.222202+0000","last_clean":"2024-03-16T20:11:49.222202+0000","last_became_active":"2024-03-16T19:56:33.970723+0000","last_became_peered":"2024-03-16T19:56:33.970723+0000","last_unstale":"2024-03-16T20:11:49.222202+0000","last_undegraded":"2024-03-16T20:11:49.222202+0000","last_fullsized":"2024-03-16T20:11:49.222202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'20","last_scrub_stamp":"2024-03-16T20:11:49.222140+0000","last_deep_scrub":"32'20","last_deep_scrub_stamp":"2024-03-16T20:11:49.222140+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.222140+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:42:28.648561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.004808+0000","last_change":"2024-03-16T20:11:56.004850+0000","last_active":"2024-03-16T20:11:56.004808+0000","last_peered":"2024-03-16T20:11:56.004808+0000","last_clean":"2024-03-16T20:11:56.004808+0000","last_became_active":"2024-03-16T19:56:43.439192+0000","last_became_peered":"2024-03-16T19:56:43.439192+0000","last_unstale":"2024-03-16T20:11:56.004808+0000","last_undegraded":"2024-03-16T20:11:56.004808+0000","last_fullsized":"2024-03-16T20:11:56.004808+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.004687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.004687+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.004687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:13:10.810213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.985844+0000","last_change":"2024-03-16T20:11:48.985887+0000","last_active":"2024-03-16T20:11:48.985844+0000","last_peered":"2024-03-16T20:11:48.985844+0000","last_clean":"2024-03-16T20:11:48.985844+0000","last_became_active":"2024-03-16T19:56:36.249185+0000","last_became_peered":"2024-03-16T19:56:36.249185+0000","last_unstale":"2024-03-16T20:11:48.985844+0000","last_undegraded":"2024-03-16T20:11:48.985844+0000","last_fullsized":"2024-03-16T20:11:48.985844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.985737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.985737+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.985737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:15.149371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"32'9","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.098421+0000","last_change":"2024-03-16T20:11:50.098471+0000","last_active":"2024-03-16T20:11:50.098421+0000","last_peered":"2024-03-16T20:11:50.098421+0000","last_clean":"2024-03-16T20:11:50.098421+0000","last_became_active":"2024-03-16T19:56:33.982704+0000","last_became_peered":"2024-03-16T19:56:33.982704+0000","last_unstale":"2024-03-16T20:11:50.098421+0000","last_undegraded":"2024-03-16T20:11:50.098421+0000","last_fullsized":"2024-03-16T20:11:50.098421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'9","last_scrub_stamp":"2024-03-16T20:11:50.098317+0000","last_deep_scrub":"32'9","last_deep_scrub_stamp":"2024-03-16T20:11:50.098317+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.098317+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:50.425995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.986198+0000","last_change":"2024-03-16T20:11:53.986252+0000","last_active":"2024-03-16T20:11:53.986198+0000","last_peered":"2024-03-16T20:11:53.986198+0000","last_clean":"2024-03-16T20:11:53.986198+0000","last_became_active":"2024-03-16T19:56:43.455531+0000","last_became_peered":"2024-03-16T19:56:43.455531+0000","last_unstale":"2024-03-16T20:11:53.986198+0000","last_undegraded":"2024-03-16T20:11:53.986198+0000","last_fullsized":"2024-03-16T20:11:53.986198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.986036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.986036+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.986036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:00.681893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.766369+0000","last_change":"2024-03-16T20:11:45.766407+0000","last_active":"2024-03-16T20:11:45.766369+0000","last_peered":"2024-03-16T20:11:45.766369+0000","last_clean":"2024-03-16T20:11:45.766369+0000","last_became_active":"2024-03-16T19:56:36.302600+0000","last_became_peered":"2024-03-16T19:56:36.302600+0000","last_unstale":"2024-03-16T20:11:45.766369+0000","last_undegraded":"2024-03-16T20:11:45.766369+0000","last_fullsized":"2024-03-16T20:11:45.766369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.766246+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.766246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:05.475898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"32'6","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.216678+0000","last_change":"2024-03-16T20:11:54.216728+0000","last_active":"2024-03-16T20:11:54.216678+0000","last_peered":"2024-03-16T20:11:54.216678+0000","last_clean":"2024-03-16T20:11:54.216678+0000","last_became_active":"2024-03-16T19:56:33.978581+0000","last_became_peered":"2024-03-16T19:56:33.978581+0000","last_unstale":"2024-03-16T20:11:54.216678+0000","last_undegraded":"2024-03-16T20:11:54.216678+0000","last_fullsized":"2024-03-16T20:11:54.216678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-03-16T20:11:54.216566+0000","last_deep_scrub":"32'6","last_deep_scrub_stamp":"2024-03-16T20:11:54.216566+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.216566+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:01:09.020994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:04.862515+0000","last_change":"2024-03-16T20:12:04.862554+0000","last_active":"2024-03-16T20:12:04.862515+0000","last_peered":"2024-03-16T20:12:04.862515+0000","last_clean":"2024-03-16T20:12:04.862515+0000","last_became_active":"2024-03-16T19:57:41.245660+0000","last_became_peered":"2024-03-16T19:57:41.245660+0000","last_unstale":"2024-03-16T20:12:04.862515+0000","last_undegraded":"2024-03-16T20:12:04.862515+0000","last_fullsized":"2024-03-16T20:12:04.862515+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:04.862404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:04.862404+0000","last_clean_scrub_stamp":"2024-03-16T20:12:04.862404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:12.258860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.141416+0000","last_change":"2024-03-16T20:12:01.141536+0000","last_active":"2024-03-16T20:12:01.141416+0000","last_peered":"2024-03-16T20:12:01.141416+0000","last_clean":"2024-03-16T20:12:01.141416+0000","last_became_active":"2024-03-16T19:56:36.318054+0000","last_became_peered":"2024-03-16T19:56:36.318054+0000","last_unstale":"2024-03-16T20:12:01.141416+0000","last_undegraded":"2024-03-16T20:12:01.141416+0000","last_fullsized":"2024-03-16T20:12:01.141416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.141112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.141112+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.141112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:08:42.178461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"32'7","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.134000+0000","last_change":"2024-03-16T20:11:53.134050+0000","last_active":"2024-03-16T20:11:53.134000+0000","last_peered":"2024-03-16T20:11:53.134000+0000","last_clean":"2024-03-16T20:11:53.134000+0000","last_became_active":"2024-03-16T19:56:33.981618+0000","last_became_peered":"2024-03-16T19:56:33.981618+0000","last_unstale":"2024-03-16T20:11:53.134000+0000","last_undegraded":"2024-03-16T20:11:53.134000+0000","last_fullsized":"2024-03-16T20:11:53.134000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:53.133903+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:53.133903+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.133903+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:54:11.540926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:01.866641+0000","last_change":"2024-03-16T20:12:01.866681+0000","last_active":"2024-03-16T20:12:01.866641+0000","last_peered":"2024-03-16T20:12:01.866641+0000","last_clean":"2024-03-16T20:12:01.866641+0000","last_became_active":"2024-03-16T19:56:43.493213+0000","last_became_peered":"2024-03-16T19:56:43.493213+0000","last_unstale":"2024-03-16T20:12:01.866641+0000","last_undegraded":"2024-03-16T20:12:01.866641+0000","last_fullsized":"2024-03-16T20:12:01.866641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:01.866518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:01.866518+0000","last_clean_scrub_stamp":"2024-03-16T20:12:01.866518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:51:13.427578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.863668+0000","last_change":"2024-03-16T20:11:56.863707+0000","last_active":"2024-03-16T20:11:56.863668+0000","last_peered":"2024-03-16T20:11:56.863668+0000","last_clean":"2024-03-16T20:11:56.863668+0000","last_became_active":"2024-03-16T19:56:43.491750+0000","last_became_peered":"2024-03-16T19:56:43.491750+0000","last_unstale":"2024-03-16T20:11:56.863668+0000","last_undegraded":"2024-03-16T20:11:56.863668+0000","last_fullsized":"2024-03-16T20:11:56.863668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.863556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.863556+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.863556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:21.840144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.082943+0000","last_change":"2024-03-16T20:11:51.082979+0000","last_active":"2024-03-16T20:11:51.082943+0000","last_peered":"2024-03-16T20:11:51.082943+0000","last_clean":"2024-03-16T20:11:51.082943+0000","last_became_active":"2024-03-16T19:56:36.221953+0000","last_became_peered":"2024-03-16T19:56:36.221953+0000","last_unstale":"2024-03-16T20:11:51.082943+0000","last_undegraded":"2024-03-16T20:11:51.082943+0000","last_fullsized":"2024-03-16T20:11:51.082943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.082823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.082823+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.082823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:06.361861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"32'4","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.213556+0000","last_change":"2024-03-16T20:11:53.213671+0000","last_active":"2024-03-16T20:11:53.213556+0000","last_peered":"2024-03-16T20:11:53.213556+0000","last_clean":"2024-03-16T20:11:53.213556+0000","last_became_active":"2024-03-16T19:56:33.976153+0000","last_became_peered":"2024-03-16T19:56:33.976153+0000","last_unstale":"2024-03-16T20:11:53.213556+0000","last_undegraded":"2024-03-16T20:11:53.213556+0000","last_fullsized":"2024-03-16T20:11:53.213556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4","last_scrub_stamp":"2024-03-16T20:11:53.213289+0000","last_deep_scrub":"32'4","last_deep_scrub_stamp":"2024-03-16T20:11:53.213289+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.213289+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:54.080243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.170593+0000","last_change":"2024-03-16T20:12:03.170618+0000","last_active":"2024-03-16T20:12:03.170593+0000","last_peered":"2024-03-16T20:12:03.170593+0000","last_clean":"2024-03-16T20:12:03.170593+0000","last_became_active":"2024-03-16T19:57:41.279282+0000","last_became_peered":"2024-03-16T19:57:41.279282+0000","last_unstale":"2024-03-16T20:12:03.170593+0000","last_undegraded":"2024-03-16T20:12:03.170593+0000","last_fullsized":"2024-03-16T20:12:03.170593+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.170526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.170526+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.170526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:28.147113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:45.883135+0000","last_change":"2024-03-16T20:11:45.883182+0000","last_active":"2024-03-16T20:11:45.883135+0000","last_peered":"2024-03-16T20:11:45.883135+0000","last_clean":"2024-03-16T20:11:45.883135+0000","last_became_active":"2024-03-16T19:56:36.314396+0000","last_became_peered":"2024-03-16T19:56:36.314396+0000","last_unstale":"2024-03-16T20:11:45.883135+0000","last_undegraded":"2024-03-16T20:11:45.883135+0000","last_fullsized":"2024-03-16T20:11:45.883135+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:45.883021+0000","last_clean_scrub_stamp":"2024-03-16T20:11:45.883021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:48.411583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"32'5","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.878274+0000","last_change":"2024-03-16T20:11:51.878321+0000","last_active":"2024-03-16T20:11:51.878274+0000","last_peered":"2024-03-16T20:11:51.878274+0000","last_clean":"2024-03-16T20:11:51.878274+0000","last_became_active":"2024-03-16T19:56:33.973325+0000","last_became_peered":"2024-03-16T19:56:33.973325+0000","last_unstale":"2024-03-16T20:11:51.878274+0000","last_undegraded":"2024-03-16T20:11:51.878274+0000","last_fullsized":"2024-03-16T20:11:51.878274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5","last_scrub_stamp":"2024-03-16T20:11:51.878180+0000","last_deep_scrub":"32'5","last_deep_scrub_stamp":"2024-03-16T20:11:51.878180+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.878180+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:58:33.086060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"32'10","reported_seq":60,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.147500+0000","last_change":"2024-03-16T20:11:51.147557+0000","last_active":"2024-03-16T20:11:51.147500+0000","last_peered":"2024-03-16T20:11:51.147500+0000","last_clean":"2024-03-16T20:11:51.147500+0000","last_became_active":"2024-03-16T19:56:33.973295+0000","last_became_peered":"2024-03-16T19:56:33.973295+0000","last_unstale":"2024-03-16T20:11:51.147500+0000","last_undegraded":"2024-03-16T20:11:51.147500+0000","last_fullsized":"2024-03-16T20:11:51.147500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'10","last_scrub_stamp":"2024-03-16T20:11:51.147398+0000","last_deep_scrub":"32'10","last_deep_scrub_stamp":"2024-03-16T20:11:51.147398+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.147398+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:56.611534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:47.819175+0000","last_change":"2024-03-16T20:11:47.819214+0000","last_active":"2024-03-16T20:11:47.819175+0000","last_peered":"2024-03-16T20:11:47.819175+0000","last_clean":"2024-03-16T20:11:47.819175+0000","last_became_active":"2024-03-16T19:56:36.305098+0000","last_became_peered":"2024-03-16T19:56:36.305098+0000","last_unstale":"2024-03-16T20:11:47.819175+0000","last_undegraded":"2024-03-16T20:11:47.819175+0000","last_fullsized":"2024-03-16T20:11:47.819175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:47.819062+0000","last_clean_scrub_stamp":"2024-03-16T20:11:47.819062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:27:50.347624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.818673+0000","last_change":"2024-03-16T20:11:50.818792+0000","last_active":"2024-03-16T20:11:50.818673+0000","last_peered":"2024-03-16T20:11:50.818673+0000","last_clean":"2024-03-16T20:11:50.818673+0000","last_became_active":"2024-03-16T19:56:43.465739+0000","last_became_peered":"2024-03-16T19:56:43.465739+0000","last_unstale":"2024-03-16T20:11:50.818673+0000","last_undegraded":"2024-03-16T20:11:50.818673+0000","last_fullsized":"2024-03-16T20:11:50.818673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.818443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.818443+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.818443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:31.275107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"32'7","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.850558+0000","last_change":"2024-03-16T20:11:49.850605+0000","last_active":"2024-03-16T20:11:49.850558+0000","last_peered":"2024-03-16T20:11:49.850558+0000","last_clean":"2024-03-16T20:11:49.850558+0000","last_became_active":"2024-03-16T19:56:33.904298+0000","last_became_peered":"2024-03-16T19:56:33.904298+0000","last_unstale":"2024-03-16T20:11:49.850558+0000","last_undegraded":"2024-03-16T20:11:49.850558+0000","last_fullsized":"2024-03-16T20:11:49.850558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7","last_scrub_stamp":"2024-03-16T20:11:49.850461+0000","last_deep_scrub":"32'7","last_deep_scrub_stamp":"2024-03-16T20:11:49.850461+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.850461+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:19.717407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.023506+0000","last_change":"2024-03-16T20:11:48.023544+0000","last_active":"2024-03-16T20:11:48.023506+0000","last_peered":"2024-03-16T20:11:48.023506+0000","last_clean":"2024-03-16T20:11:48.023506+0000","last_became_active":"2024-03-16T19:56:36.247357+0000","last_became_peered":"2024-03-16T19:56:36.247357+0000","last_unstale":"2024-03-16T20:11:48.023506+0000","last_undegraded":"2024-03-16T20:11:48.023506+0000","last_fullsized":"2024-03-16T20:11:48.023506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.023400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.023400+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.023400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:17.327792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.870788+0000","last_change":"2024-03-16T20:11:57.870832+0000","last_active":"2024-03-16T20:11:57.870788+0000","last_peered":"2024-03-16T20:11:57.870788+0000","last_clean":"2024-03-16T20:11:57.870788+0000","last_became_active":"2024-03-16T19:56:43.463494+0000","last_became_peered":"2024-03-16T19:56:43.463494+0000","last_unstale":"2024-03-16T20:11:57.870788+0000","last_undegraded":"2024-03-16T20:11:57.870788+0000","last_fullsized":"2024-03-16T20:11:57.870788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.870655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.870655+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.870655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:56.093848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.212579+0000","last_change":"2024-03-16T20:12:00.212616+0000","last_active":"2024-03-16T20:12:00.212579+0000","last_peered":"2024-03-16T20:12:00.212579+0000","last_clean":"2024-03-16T20:12:00.212579+0000","last_became_active":"2024-03-16T19:56:36.317882+0000","last_became_peered":"2024-03-16T19:56:36.317882+0000","last_unstale":"2024-03-16T20:12:00.212579+0000","last_undegraded":"2024-03-16T20:12:00.212579+0000","last_fullsized":"2024-03-16T20:12:00.212579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.212475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.212475+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.212475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:58:41.420354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.876522+0000","last_change":"2024-03-16T20:11:48.876540+0000","last_active":"2024-03-16T20:11:48.876522+0000","last_peered":"2024-03-16T20:11:48.876522+0000","last_clean":"2024-03-16T20:11:48.876522+0000","last_became_active":"2024-03-16T19:56:36.298425+0000","last_became_peered":"2024-03-16T19:56:36.298425+0000","last_unstale":"2024-03-16T20:11:48.876522+0000","last_undegraded":"2024-03-16T20:11:48.876522+0000","last_fullsized":"2024-03-16T20:11:48.876522+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.876460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.876460+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.876460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:15.040094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.277449+0000","last_change":"2024-03-16T20:11:49.277493+0000","last_active":"2024-03-16T20:11:49.277449+0000","last_peered":"2024-03-16T20:11:49.277449+0000","last_clean":"2024-03-16T20:11:49.277449+0000","last_became_active":"2024-03-16T19:56:36.306109+0000","last_became_peered":"2024-03-16T19:56:36.306109+0000","last_unstale":"2024-03-16T20:11:49.277449+0000","last_undegraded":"2024-03-16T20:11:49.277449+0000","last_fullsized":"2024-03-16T20:11:49.277449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.277321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.277321+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.277321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:52:08.986973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.873763+0000","last_change":"2024-03-16T20:11:51.873803+0000","last_active":"2024-03-16T20:11:51.873763+0000","last_peered":"2024-03-16T20:11:51.873763+0000","last_clean":"2024-03-16T20:11:51.873763+0000","last_became_active":"2024-03-16T19:56:36.317097+0000","last_became_peered":"2024-03-16T19:56:36.317097+0000","last_unstale":"2024-03-16T20:11:51.873763+0000","last_undegraded":"2024-03-16T20:11:51.873763+0000","last_fullsized":"2024-03-16T20:11:51.873763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.873657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.873657+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.873657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:21.740603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:53.877090+0000","last_change":"2024-03-16T20:11:53.877131+0000","last_active":"2024-03-16T20:11:53.877090+0000","last_peered":"2024-03-16T20:11:53.877090+0000","last_clean":"2024-03-16T20:11:53.877090+0000","last_became_active":"2024-03-16T19:56:36.255125+0000","last_became_peered":"2024-03-16T19:56:36.255125+0000","last_unstale":"2024-03-16T20:11:53.877090+0000","last_undegraded":"2024-03-16T20:11:53.877090+0000","last_fullsized":"2024-03-16T20:11:53.877090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:53.876983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:53.876983+0000","last_clean_scrub_stamp":"2024-03-16T20:11:53.876983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:02:40.995365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.058999+0000","last_change":"2024-03-16T20:11:52.059035+0000","last_active":"2024-03-16T20:11:52.058999+0000","last_peered":"2024-03-16T20:11:52.058999+0000","last_clean":"2024-03-16T20:11:52.058999+0000","last_became_active":"2024-03-16T19:56:36.222094+0000","last_became_peered":"2024-03-16T19:56:36.222094+0000","last_unstale":"2024-03-16T20:11:52.058999+0000","last_undegraded":"2024-03-16T20:11:52.058999+0000","last_fullsized":"2024-03-16T20:11:52.058999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.058883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.058883+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.058883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:23.372950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.178239+0000","last_change":"2024-03-16T20:11:59.178344+0000","last_active":"2024-03-16T20:11:59.178239+0000","last_peered":"2024-03-16T20:11:59.178239+0000","last_clean":"2024-03-16T20:11:59.178239+0000","last_became_active":"2024-03-16T19:56:36.317722+0000","last_became_peered":"2024-03-16T19:56:36.317722+0000","last_unstale":"2024-03-16T20:11:59.178239+0000","last_undegraded":"2024-03-16T20:11:59.178239+0000","last_fullsized":"2024-03-16T20:11:59.178239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.177970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.177970+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.177970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:14.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.858816+0000","last_change":"2024-03-16T20:11:54.858910+0000","last_active":"2024-03-16T20:11:54.858816+0000","last_peered":"2024-03-16T20:11:54.858816+0000","last_clean":"2024-03-16T20:11:54.858816+0000","last_became_active":"2024-03-16T19:56:36.256407+0000","last_became_peered":"2024-03-16T19:56:36.256407+0000","last_unstale":"2024-03-16T20:11:54.858816+0000","last_undegraded":"2024-03-16T20:11:54.858816+0000","last_fullsized":"2024-03-16T20:11:54.858816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.858562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.858562+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.858562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:19.835150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.181761+0000","last_change":"2024-03-16T20:11:55.181799+0000","last_active":"2024-03-16T20:11:55.181761+0000","last_peered":"2024-03-16T20:11:55.181761+0000","last_clean":"2024-03-16T20:11:55.181761+0000","last_became_active":"2024-03-16T19:56:36.293084+0000","last_became_peered":"2024-03-16T19:56:36.293084+0000","last_unstale":"2024-03-16T20:11:55.181761+0000","last_undegraded":"2024-03-16T20:11:55.181761+0000","last_fullsized":"2024-03-16T20:11:55.181761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.181654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.181654+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.181654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:47.930472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.897835+0000","last_change":"2024-03-16T20:11:55.897835+0000","last_active":"2024-03-16T20:11:55.897835+0000","last_peered":"2024-03-16T20:11:55.897835+0000","last_clean":"2024-03-16T20:11:55.897835+0000","last_became_active":"2024-03-16T19:56:36.256650+0000","last_became_peered":"2024-03-16T19:56:36.256650+0000","last_unstale":"2024-03-16T20:11:55.897835+0000","last_undegraded":"2024-03-16T20:11:55.897835+0000","last_fullsized":"2024-03-16T20:11:55.897835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.897455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.897455+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.897455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:22.139684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:00.166087+0000","last_change":"2024-03-16T20:12:00.166087+0000","last_active":"2024-03-16T20:12:00.166087+0000","last_peered":"2024-03-16T20:12:00.166087+0000","last_clean":"2024-03-16T20:12:00.166087+0000","last_became_active":"2024-03-16T19:56:36.317238+0000","last_became_peered":"2024-03-16T19:56:36.317238+0000","last_unstale":"2024-03-16T20:12:00.166087+0000","last_undegraded":"2024-03-16T20:12:00.166087+0000","last_fullsized":"2024-03-16T20:12:00.166087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:00.165933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:00.165933+0000","last_clean_scrub_stamp":"2024-03-16T20:12:00.165933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:26.408162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:52.867262+0000","last_change":"2024-03-16T20:11:52.867262+0000","last_active":"2024-03-16T20:11:52.867262+0000","last_peered":"2024-03-16T20:11:52.867262+0000","last_clean":"2024-03-16T20:11:52.867262+0000","last_became_active":"2024-03-16T19:56:36.323694+0000","last_became_peered":"2024-03-16T19:56:36.323694+0000","last_unstale":"2024-03-16T20:11:52.867262+0000","last_undegraded":"2024-03-16T20:11:52.867262+0000","last_fullsized":"2024-03-16T20:11:52.867262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:52.867111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:52.867111+0000","last_clean_scrub_stamp":"2024-03-16T20:11:52.867111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:08.289839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.145536+0000","last_change":"2024-03-16T20:11:56.145536+0000","last_active":"2024-03-16T20:11:56.145536+0000","last_peered":"2024-03-16T20:11:56.145536+0000","last_clean":"2024-03-16T20:11:56.145536+0000","last_became_active":"2024-03-16T19:56:36.304061+0000","last_became_peered":"2024-03-16T19:56:36.304061+0000","last_unstale":"2024-03-16T20:11:56.145536+0000","last_undegraded":"2024-03-16T20:11:56.145536+0000","last_fullsized":"2024-03-16T20:11:56.145536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.145390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.145390+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.145390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:51:51.466788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:50.275299+0000","last_change":"2024-03-16T20:11:50.275299+0000","last_active":"2024-03-16T20:11:50.275299+0000","last_peered":"2024-03-16T20:11:50.275299+0000","last_clean":"2024-03-16T20:11:50.275299+0000","last_became_active":"2024-03-16T19:56:36.231892+0000","last_became_peered":"2024-03-16T19:56:36.231892+0000","last_unstale":"2024-03-16T20:11:50.275299+0000","last_undegraded":"2024-03-16T20:11:50.275299+0000","last_fullsized":"2024-03-16T20:11:50.275299+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:50.274977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:50.274977+0000","last_clean_scrub_stamp":"2024-03-16T20:11:50.274977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:55.739112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.849641+0000","last_change":"2024-03-16T20:11:49.849679+0000","last_active":"2024-03-16T20:11:49.849641+0000","last_peered":"2024-03-16T20:11:49.849641+0000","last_clean":"2024-03-16T20:11:49.849641+0000","last_became_active":"2024-03-16T19:56:36.309399+0000","last_became_peered":"2024-03-16T19:56:36.309399+0000","last_unstale":"2024-03-16T20:11:49.849641+0000","last_undegraded":"2024-03-16T20:11:49.849641+0000","last_fullsized":"2024-03-16T20:11:49.849641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.849534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.849534+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.849534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:12:20.595436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:03.183023+0000","last_change":"2024-03-16T20:12:03.183057+0000","last_active":"2024-03-16T20:12:03.183023+0000","last_peered":"2024-03-16T20:12:03.183023+0000","last_clean":"2024-03-16T20:12:03.183023+0000","last_became_active":"2024-03-16T19:56:36.318927+0000","last_became_peered":"2024-03-16T19:56:36.318927+0000","last_unstale":"2024-03-16T20:12:03.183023+0000","last_undegraded":"2024-03-16T20:12:03.183023+0000","last_fullsized":"2024-03-16T20:12:03.183023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:03.182944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:03.182944+0000","last_clean_scrub_stamp":"2024-03-16T20:12:03.182944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:56:14.282067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:59.178440+0000","last_change":"2024-03-16T20:11:59.178476+0000","last_active":"2024-03-16T20:11:59.178440+0000","last_peered":"2024-03-16T20:11:59.178440+0000","last_clean":"2024-03-16T20:11:59.178440+0000","last_became_active":"2024-03-16T19:56:36.316347+0000","last_became_peered":"2024-03-16T19:56:36.316347+0000","last_unstale":"2024-03-16T20:11:59.178440+0000","last_undegraded":"2024-03-16T20:11:59.178440+0000","last_fullsized":"2024-03-16T20:11:59.178440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:59.178344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:59.178344+0000","last_clean_scrub_stamp":"2024-03-16T20:11:59.178344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:15:24.154931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.058485+0000","last_change":"2024-03-16T20:11:54.058540+0000","last_active":"2024-03-16T20:11:54.058485+0000","last_peered":"2024-03-16T20:11:54.058485+0000","last_clean":"2024-03-16T20:11:54.058485+0000","last_became_active":"2024-03-16T19:56:36.253618+0000","last_became_peered":"2024-03-16T19:56:36.253618+0000","last_unstale":"2024-03-16T20:11:54.058485+0000","last_undegraded":"2024-03-16T20:11:54.058485+0000","last_fullsized":"2024-03-16T20:11:54.058485+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.058308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.058308+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.058308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:58.415808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:49.834025+0000","last_change":"2024-03-16T20:11:49.834066+0000","last_active":"2024-03-16T20:11:49.834025+0000","last_peered":"2024-03-16T20:11:49.834025+0000","last_clean":"2024-03-16T20:11:49.834025+0000","last_became_active":"2024-03-16T19:56:36.303022+0000","last_became_peered":"2024-03-16T19:56:36.303022+0000","last_unstale":"2024-03-16T20:11:49.834025+0000","last_undegraded":"2024-03-16T20:11:49.834025+0000","last_fullsized":"2024-03-16T20:11:49.834025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:49.833918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:49.833918+0000","last_clean_scrub_stamp":"2024-03-16T20:11:49.833918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:50.700873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:56.864092+0000","last_change":"2024-03-16T20:11:56.864132+0000","last_active":"2024-03-16T20:11:56.864092+0000","last_peered":"2024-03-16T20:11:56.864092+0000","last_clean":"2024-03-16T20:11:56.864092+0000","last_became_active":"2024-03-16T19:56:36.301584+0000","last_became_peered":"2024-03-16T19:56:36.301584+0000","last_unstale":"2024-03-16T20:11:56.864092+0000","last_undegraded":"2024-03-16T20:11:56.864092+0000","last_fullsized":"2024-03-16T20:11:56.864092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:56.863991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:56.863991+0000","last_clean_scrub_stamp":"2024-03-16T20:11:56.863991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:08:37.901340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.977309+0000","last_change":"2024-03-16T20:11:51.977345+0000","last_active":"2024-03-16T20:11:51.977309+0000","last_peered":"2024-03-16T20:11:51.977309+0000","last_clean":"2024-03-16T20:11:51.977309+0000","last_became_active":"2024-03-16T19:56:36.304947+0000","last_became_peered":"2024-03-16T19:56:36.304947+0000","last_unstale":"2024-03-16T20:11:51.977309+0000","last_undegraded":"2024-03-16T20:11:51.977309+0000","last_fullsized":"2024-03-16T20:11:51.977309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.977190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.977190+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.977190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:13.226901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:54.161201+0000","last_change":"2024-03-16T20:11:54.161238+0000","last_active":"2024-03-16T20:11:54.161201+0000","last_peered":"2024-03-16T20:11:54.161201+0000","last_clean":"2024-03-16T20:11:54.161201+0000","last_became_active":"2024-03-16T19:56:36.298127+0000","last_became_peered":"2024-03-16T19:56:36.298127+0000","last_unstale":"2024-03-16T20:11:54.161201+0000","last_undegraded":"2024-03-16T20:11:54.161201+0000","last_fullsized":"2024-03-16T20:11:54.161201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:54.161095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:54.161095+0000","last_clean_scrub_stamp":"2024-03-16T20:11:54.161095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:11:06.244614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:57.188449+0000","last_change":"2024-03-16T20:11:57.188489+0000","last_active":"2024-03-16T20:11:57.188449+0000","last_peered":"2024-03-16T20:11:57.188449+0000","last_clean":"2024-03-16T20:11:57.188449+0000","last_became_active":"2024-03-16T19:56:36.303503+0000","last_became_peered":"2024-03-16T19:56:36.303503+0000","last_unstale":"2024-03-16T20:11:57.188449+0000","last_undegraded":"2024-03-16T20:11:57.188449+0000","last_fullsized":"2024-03-16T20:11:57.188449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:57.188342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:57.188342+0000","last_clean_scrub_stamp":"2024-03-16T20:11:57.188342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:10.191861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:06.477630+0000","last_change":"2024-03-16T20:12:06.477684+0000","last_active":"2024-03-16T20:12:06.477630+0000","last_peered":"2024-03-16T20:12:06.477630+0000","last_clean":"2024-03-16T20:12:06.477630+0000","last_became_active":"2024-03-16T19:56:36.298840+0000","last_became_peered":"2024-03-16T19:56:36.298840+0000","last_unstale":"2024-03-16T20:12:06.477630+0000","last_undegraded":"2024-03-16T20:12:06.477630+0000","last_fullsized":"2024-03-16T20:12:06.477630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:06.477451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:06.477451+0000","last_clean_scrub_stamp":"2024-03-16T20:12:06.477451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:08:11.396502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:48.840694+0000","last_change":"2024-03-16T20:11:48.840737+0000","last_active":"2024-03-16T20:11:48.840694+0000","last_peered":"2024-03-16T20:11:48.840694+0000","last_clean":"2024-03-16T20:11:48.840694+0000","last_became_active":"2024-03-16T19:56:36.300411+0000","last_became_peered":"2024-03-16T19:56:36.300411+0000","last_unstale":"2024-03-16T20:11:48.840694+0000","last_undegraded":"2024-03-16T20:11:48.840694+0000","last_fullsized":"2024-03-16T20:11:48.840694+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:48.840578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:48.840578+0000","last_clean_scrub_stamp":"2024-03-16T20:11:48.840578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:01.844097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:04.440300+0000","last_change":"2024-03-16T20:12:04.440431+0000","last_active":"2024-03-16T20:12:04.440300+0000","last_peered":"2024-03-16T20:12:04.440300+0000","last_clean":"2024-03-16T20:12:04.440300+0000","last_became_active":"2024-03-16T19:56:36.296618+0000","last_became_peered":"2024-03-16T19:56:36.296618+0000","last_unstale":"2024-03-16T20:12:04.440300+0000","last_undegraded":"2024-03-16T20:12:04.440300+0000","last_fullsized":"2024-03-16T20:12:04.440300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:04.440076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:04.440076+0000","last_clean_scrub_stamp":"2024-03-16T20:12:04.440076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:57.132176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:02.189723+0000","last_change":"2024-03-16T20:12:02.189759+0000","last_active":"2024-03-16T20:12:02.189723+0000","last_peered":"2024-03-16T20:12:02.189723+0000","last_clean":"2024-03-16T20:12:02.189723+0000","last_became_active":"2024-03-16T19:56:36.317741+0000","last_became_peered":"2024-03-16T19:56:36.317741+0000","last_unstale":"2024-03-16T20:12:02.189723+0000","last_undegraded":"2024-03-16T20:12:02.189723+0000","last_fullsized":"2024-03-16T20:12:02.189723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:02.189621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:02.189621+0000","last_clean_scrub_stamp":"2024-03-16T20:12:02.189621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:00.412815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.301282+0000","last_change":"2024-03-16T20:11:51.301376+0000","last_active":"2024-03-16T20:11:51.301282+0000","last_peered":"2024-03-16T20:11:51.301282+0000","last_clean":"2024-03-16T20:11:51.301282+0000","last_became_active":"2024-03-16T19:56:36.231671+0000","last_became_peered":"2024-03-16T19:56:36.231671+0000","last_unstale":"2024-03-16T20:11:51.301282+0000","last_undegraded":"2024-03-16T20:11:51.301282+0000","last_fullsized":"2024-03-16T20:11:51.301282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.301088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.301088+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.301088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:02:19.083559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:51.797896+0000","last_change":"2024-03-16T20:11:51.797931+0000","last_active":"2024-03-16T20:11:51.797896+0000","last_peered":"2024-03-16T20:11:51.797896+0000","last_clean":"2024-03-16T20:11:51.797896+0000","last_became_active":"2024-03-16T19:57:41.236104+0000","last_became_peered":"2024-03-16T19:57:41.236104+0000","last_unstale":"2024-03-16T20:11:51.797896+0000","last_undegraded":"2024-03-16T20:11:51.797896+0000","last_fullsized":"2024-03-16T20:11:51.797896+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:51.797788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:51.797788+0000","last_clean_scrub_stamp":"2024-03-16T20:11:51.797788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:54:13.047499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:44.718717+0000","last_change":"2024-03-16T20:11:44.718754+0000","last_active":"2024-03-16T20:11:44.718717+0000","last_peered":"2024-03-16T20:11:44.718717+0000","last_clean":"2024-03-16T20:11:44.718717+0000","last_became_active":"2024-03-16T19:56:36.302719+0000","last_became_peered":"2024-03-16T19:56:36.302719+0000","last_unstale":"2024-03-16T20:11:44.718717+0000","last_undegraded":"2024-03-16T20:11:44.718717+0000","last_fullsized":"2024-03-16T20:11:44.718717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:44.718600+0000","last_clean_scrub_stamp":"2024-03-16T20:11:44.718600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:50.182736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:11:55.015497+0000","last_change":"2024-03-16T20:11:55.015535+0000","last_active":"2024-03-16T20:11:55.015497+0000","last_peered":"2024-03-16T20:11:55.015497+0000","last_clean":"2024-03-16T20:11:55.015497+0000","last_became_active":"2024-03-16T19:56:36.315522+0000","last_became_peered":"2024-03-16T19:56:36.315522+0000","last_unstale":"2024-03-16T20:11:55.015497+0000","last_undegraded":"2024-03-16T20:11:55.015497+0000","last_fullsized":"2024-03-16T20:11:55.015497+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:11:55.015394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:11:55.015394+0000","last_clean_scrub_stamp":"2024-03-16T20:11:55.015394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:35.056024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-16T20:12:07.448221+0000","last_change":"2024-03-16T20:12:07.448255+0000","last_active":"2024-03-16T20:12:07.448221+0000","last_peered":"2024-03-16T20:12:07.448221+0000","last_clean":"2024-03-16T20:12:07.448221+0000","last_became_active":"2024-03-16T19:56:36.301517+0000","last_became_peered":"2024-03-16T19:56:36.301517+0000","last_unstale":"2024-03-16T20:12:07.448221+0000","last_undegraded":"2024-03-16T20:12:07.448221+0000","last_fullsized":"2024-03-16T20:12:07.448221+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:12:07.448112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:12:07.448112+0000","last_clean_scrub_stamp":"2024-03-16T20:12:07.448112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:12:21.075777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":208,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476940,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":736918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476942,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476941,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":389120,"data_stored":148382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476939,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":145780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":149116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":372736,"data_stored":145912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":962560,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542348,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":368640,"data_stored":145982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:12:31.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-16T20:12:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.739+0000 7f38d41b3640 1 Processor -- start 2024-03-16T20:12:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.739+0000 7f38d41b3640 1 -- start start 2024-03-16T20:12:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc1541a0 0x7f38cc06b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38cc06ba90 0x7f38cc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc159e30 con 0x7f38cc15a7c0 2024-03-16T20:12:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc15ac50 con 0x7f38cc06ba90 2024-03-16T20:12:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d41b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc154580 con 0x7f38cc1541a0 2024-03-16T20:12:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d2750640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d2750640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55324/0 (socket says 172.21.15.32:55324) 2024-03-16T20:12:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d2750640 1 -- 172.21.15.32:0/414920809 learned_addr learned my addr 172.21.15.32:0/414920809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:12:31.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.743+0000 7f38d174e640 1 --2- 172.21.15.32:0/414920809 >> v2:172.21.15.59:3300/0 conn(0x7f38cc06ba90 0x7f38cc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.747+0000 7f38d2750640 1 -- 172.21.15.32:0/414920809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc1541a0 msgr2=0x7f38cc06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.747+0000 7f38d2750640 1 --2- 172.21.15.32:0/414920809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc1541a0 0x7f38cc06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.747+0000 7f38d2750640 1 -- 172.21.15.32:0/414920809 >> v2:172.21.15.59:3300/0 conn(0x7f38cc06ba90 msgr2=0x7f38cc15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:12:31.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.747+0000 7f38d2750640 1 --2- 172.21.15.32:0/414920809 >> v2:172.21.15.59:3300/0 conn(0x7f38cc06ba90 0x7f38cc15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.747+0000 7f38d2750640 1 -- 172.21.15.32:0/414920809 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc15d5c0 con 0x7f38cc15a7c0 2024-03-16T20:12:31.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.751+0000 7f38d2750640 1 --2- 172.21.15.32:0/414920809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f38cc067880 tx=0x7f38c401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701b3bf0578d2042 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:31.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d0f4d640 1 -- 172.21.15.32:0/414920809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40af400 con 0x7f38cc15a7c0 2024-03-16T20:12:31.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d0f4d640 1 -- 172.21.15.32:0/414920809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c40ab050 con 0x7f38cc15a7c0 2024-03-16T20:12:31.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d0f4d640 1 -- 172.21.15.32:0/414920809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40af780 con 0x7f38cc15a7c0 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- 172.21.15.32:0/414920809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc15a7c0 msgr2=0x7f38cc0cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 --2- 172.21.15.32:0/414920809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f38cc067880 tx=0x7f38c401f1e0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- 172.21.15.32:0/414920809 shutdown_connections 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 --2- 172.21.15.32:0/414920809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc15a7c0 0x7f38cc0cc5d0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 --2- 172.21.15.32:0/414920809 >> v2:172.21.15.59:3300/0 conn(0x7f38cc06ba90 0x7f38cc15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 --2- 172.21.15.32:0/414920809 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc1541a0 0x7f38cc06b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- 172.21.15.32:0/414920809 >> 172.21.15.32:0/414920809 conn(0x7f38cc066ea0 msgr2=0x7f38cc0721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:31.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- 172.21.15.32:0/414920809 shutdown_connections 2024-03-16T20:12:31.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- 172.21.15.32:0/414920809 wait complete. 2024-03-16T20:12:31.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 Processor -- start 2024-03-16T20:12:31.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.759+0000 7f38d41b3640 1 -- start start 2024-03-16T20:12:31.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc06ba90 0x7f38cc153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc15a7c0 0x7f38cc1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:31.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc159a70 con 0x7f38cc1541a0 2024-03-16T20:12:31.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc156050 con 0x7f38cc06ba90 2024-03-16T20:12:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d41b3640 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc142650 con 0x7f38cc15a7c0 2024-03-16T20:12:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d174e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d1f4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc06ba90 0x7f38cc153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d174e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55332/0 (socket says 172.21.15.32:55332) 2024-03-16T20:12:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d174e640 1 -- 172.21.15.32:0/29668820 learned_addr learned my addr 172.21.15.32:0/29668820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-16T20:12:31.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.763+0000 7f38d2750640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc15a7c0 0x7f38cc1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 -- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc15a7c0 msgr2=0x7f38cc1564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc15a7c0 0x7f38cc1564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 -- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc06ba90 msgr2=0x7f38cc153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc06ba90 0x7f38cc153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 -- 172.21.15.32:0/29668820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4009b30 con 0x7f38cc1541a0 2024-03-16T20:12:31.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.767+0000 7f38d174e640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c820 tx=0x7f38bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc007a30 con 0x7f38cc1541a0 2024-03-16T20:12:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc007b90 con 0x7f38cc1541a0 2024-03-16T20:12:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38d41b3640 1 -- 172.21.15.32:0/29668820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc15b4c0 con 0x7f38cc1541a0 2024-03-16T20:12:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00ce10 con 0x7f38cc1541a0 2024-03-16T20:12:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38d41b3640 1 -- 172.21.15.32:0/29668820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc15ba80 con 0x7f38cc1541a0 2024-03-16T20:12:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:31.775+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a80052a0 con 0x7f38cc1541a0 2024-03-16T20:12:32.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.083+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f38bc024480 con 0x7f38cc1541a0 2024-03-16T20:12:32.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.083+0000 7f38c27fc640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 0x7f38b0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:12:32.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.083+0000 7f38d1f4f640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 0x7f38b0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:12:32.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.087+0000 7f38d1f4f640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 0x7f38b0069380 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009990 tx=0x7f38c8009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:12:32.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.107+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f38bc055b40 con 0x7f38cc1541a0 2024-03-16T20:12:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.107+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176632 (secure 0 0 0) 0x7f38bc051de0 con 0x7f38cc1541a0 2024-03-16T20:12:32.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.379+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f38a8005550 con 0x7f38cc1541a0 2024-03-16T20:12:32.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.483+0000 7f38c27fc640 1 -- 172.21.15.32:0/29668820 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f38bc01f020 con 0x7f38cc1541a0 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 msgr2=0x7f38b0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 0x7f38b0069380 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009990 tx=0x7f38c8009110 comp rx=0 tx=0).stop 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 msgr2=0x7f38cc146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c820 tx=0x7f38bc00cce0 comp rx=0 tx=0).stop 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 shutdown_connections 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:6800/1649113964,v1:172.21.15.32:6801/1649113964] conn(0x7f38b0066ec0 0x7f38b0069380 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38cc15a7c0 0x7f38cc1564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc1541a0 0x7f38cc146220 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:32.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 --2- 172.21.15.32:0/29668820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc06ba90 0x7f38cc153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:12:32.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 >> 172.21.15.32:0/29668820 conn(0x7f38cc066ea0 msgr2=0x7f38cc0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:12:32.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 shutdown_connections 2024-03-16T20:12:32.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-16T20:12:32.487+0000 7f38affff640 1 -- 172.21.15.32:0/29668820 wait complete. 2024-03-16T20:12:32.503 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-16T20:12:32.505 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-16T20:12:32.505 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-16T20:12:32.505 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:12:43.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:12:43.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:13:08.329 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:17:52.405 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.332 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:17:52.409 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.368 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:17:52.382 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.370 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:17:52.447 27584== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.373 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:17:52.387 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.444 INFO:tasks.ceph.mon.b.smithi059.stderr:==00:00:17:52.458 27519== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.445 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:17:52.459 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.450 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:17:52.464 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:08.519 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:17:52.533 27520== Warning: unimplemented fcntl command: 1036 2024-03-16T20:13:13.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:13:13.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:13:16.713 INFO:tasks.ceph.mds.b.smithi059.stderr:2024-03-16T20:13:16.703+0000 13df2640 -1 MDSIOContextBase: failed with -108, restarting... 2024-03-16T20:13:43.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:13:43.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:14:13.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:14:13.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:14:43.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:14:43.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:14:58.352 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-03-16T20:13:55.480+0000 b9e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 31884) UID: 0 2024-03-16T20:14:58.352 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-03-16T20:13:55.484+0000 b9e2640 -1 mds.a *** got signal Terminated *** 2024-03-16T20:15:13.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:15:13.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:15:43.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:15:43.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:16:13.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:16:13.652 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:16:43.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:16:43.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:17:13.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:17:13.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:17:31.579 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-16T20:17:31.579 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-16T20:17:31.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:17:31.580 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T20:17:31.584+0000 7fd4acd90640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 27777) UID: 0 2024-03-16T20:17:31.581 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-16T20:17:31.584+0000 7fd4acd90640 -1 osd.0 32 *** Got signal Terminated *** 2024-03-16T20:17:37.680 INFO:tasks.ceph.osd.0:Stopped 2024-03-16T20:17:37.680 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-16T20:17:37.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:17:37.681 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T20:17:37.684+0000 7efee108e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 27779) UID: 0 2024-03-16T20:17:37.681 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-16T20:17:37.684+0000 7efee108e640 -1 osd.1 34 *** Got signal Terminated *** 2024-03-16T20:17:37.959 INFO:tasks.ceph.mds.b.smithi059.stderr: -1> 2024-03-16T20:13:16.703+0000 13df2640 -1 MDSIOContextBase: failed with -108, restarting... 2024-03-16T20:17:38.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-16T20:17:43.782 INFO:tasks.ceph.osd.1:Stopped 2024-03-16T20:17:43.782 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-16T20:17:43.782 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:17:43.783 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T20:17:43.784+0000 7f8822ae4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 27781) UID: 0 2024-03-16T20:17:43.783 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-16T20:17:43.784+0000 7f8822ae4640 -1 osd.2 36 *** Got signal Terminated *** 2024-03-16T20:17:43.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:17:43.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:17:45.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-16T20:17:45.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-16T20:17:49.883 INFO:tasks.ceph.osd.2:Stopped 2024-03-16T20:17:49.884 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-16T20:17:49.884 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:17:49.884 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T20:17:49.888+0000 7fa0d71b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 27783) UID: 0 2024-03-16T20:17:49.885 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-16T20:17:49.888+0000 7fa0d71b1640 -1 osd.3 38 *** Got signal Terminated *** 2024-03-16T20:17:51.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-16T20:17:55.985 INFO:tasks.ceph.osd.3:Stopped 2024-03-16T20:17:55.986 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-16T20:17:55.986 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:17:55.987 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T20:17:55.983+0000 7f9b9f665640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 27837) UID: 0 2024-03-16T20:17:55.987 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-03-16T20:17:55.983+0000 7f9b9f665640 -1 osd.4 40 *** Got signal Terminated *** 2024-03-16T20:18:02.087 INFO:tasks.ceph.osd.4:Stopped 2024-03-16T20:18:02.088 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-16T20:18:02.088 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:02.089 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T20:18:02.087+0000 7eff145cf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 27839) UID: 0 2024-03-16T20:18:02.089 INFO:tasks.ceph.osd.5.smithi059.stderr:2024-03-16T20:18:02.087+0000 7eff145cf640 -1 osd.5 42 *** Got signal Terminated *** 2024-03-16T20:18:04.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-16T20:18:04.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-16T20:18:08.190 INFO:tasks.ceph.osd.5:Stopped 2024-03-16T20:18:08.190 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-16T20:18:08.190 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:08.191 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T20:18:08.187+0000 7f8bb46e7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 27841) UID: 0 2024-03-16T20:18:08.191 INFO:tasks.ceph.osd.6.smithi059.stderr:2024-03-16T20:18:08.187+0000 7f8bb46e7640 -1 osd.6 44 *** Got signal Terminated *** 2024-03-16T20:18:10.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-16T20:18:10.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~6s 2024-03-16T20:18:10.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-16T20:18:13.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:13.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:14.292 INFO:tasks.ceph.osd.6:Stopped 2024-03-16T20:18:14.292 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-16T20:18:14.293 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:14.294 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T20:18:14.291+0000 7fc39b687640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 27843) UID: 0 2024-03-16T20:18:14.294 INFO:tasks.ceph.osd.7.smithi059.stderr:2024-03-16T20:18:14.291+0000 7fc39b687640 -1 osd.7 44 *** Got signal Terminated *** 2024-03-16T20:18:16.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-16T20:18:16.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~12s 2024-03-16T20:18:16.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-16T20:18:20.395 INFO:tasks.ceph.osd.7:Stopped 2024-03-16T20:18:20.395 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-16T20:18:20.395 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-16T20:18:20.395 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:20.396 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T20:18:20.395+0000 7f68fbca2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 27586) UID: 0 2024-03-16T20:18:20.396 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-16T20:18:20.395+0000 7f68fbca2640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T20:18:20.423 INFO:tasks.ceph.mgr.y:Stopped 2024-03-16T20:18:20.424 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-16T20:18:20.424 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:20.448 INFO:tasks.ceph.mgr.x:Stopped 2024-03-16T20:18:20.448 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-16T20:18:20.449 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-16T20:18:20.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:20.487 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-16T20:18:20.471+0000 a383640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 27581) UID: 0 2024-03-16T20:18:20.488 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-16T20:18:20.471+0000 a383640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T20:18:22.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-16T20:18:22.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~18s 2024-03-16T20:18:22.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-16T20:18:26.549 INFO:tasks.ceph.mon.a:Stopped 2024-03-16T20:18:26.550 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-16T20:18:26.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:26.590 INFO:tasks.ceph.mon.b.smithi059.stderr:2024-03-16T20:18:26.555+0000 a383640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 27512) UID: 0 2024-03-16T20:18:26.592 INFO:tasks.ceph.mon.b.smithi059.stderr:2024-03-16T20:18:26.559+0000 a383640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-16T20:18:28.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-16T20:18:28.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~24s 2024-03-16T20:18:28.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-16T20:18:32.652 INFO:tasks.ceph.mon.b:Stopped 2024-03-16T20:18:32.737 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-16T20:18:32.737 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:18:32.778 INFO:tasks.ceph.mon.c.smithi059.stderr:2024-03-16T20:18:32.743+0000 a383640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 27515) UID: 0 2024-03-16T20:18:32.779 INFO:tasks.ceph.mon.c.smithi059.stderr:2024-03-16T20:18:32.747+0000 a383640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2024-03-16T20:18:33.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~30s 2024-03-16T20:18:33.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-16T20:18:38.840 INFO:tasks.ceph.mon.c:Stopped 2024-03-16T20:18:38.840 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-16T20:18:38.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-16T20:18:38.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T20:18:38.857 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-16T20:18:39.109 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T20:18:39.109 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-16T20:18:39.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-16T20:18:39.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~35s 2024-03-16T20:18:39.259 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T20:18:39.259 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-16T20:18:39.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T20:18:39.394 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-16T20:18:39.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T20:18:39.544 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-16T20:18:39.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T20:18:39.703 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-16T20:18:40.015 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T20:18:40.015 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-16T20:18:40.181 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T20:18:40.182 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-16T20:18:40.356 INFO:tasks.ceph:Archiving mon data... 2024-03-16T20:18:40.357 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/data/mon.a.tgz 2024-03-16T20:18:40.358 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-16T20:18:40.363 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.aKj3cUKsBT 2024-03-16T20:18:40.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aKj3cUKsBT 2024-03-16T20:18:40.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.aKj3cUKsBT 2024-03-16T20:18:40.553 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.aKj3cUKsBT is 323KB 2024-03-16T20:18:40.610 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.aKj3cUKsBT 2024-03-16T20:18:40.615 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/data/mon.b.tgz 2024-03-16T20:18:40.615 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-16T20:18:40.620 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.JKSmbINum0 2024-03-16T20:18:40.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.JKSmbINum0 2024-03-16T20:18:40.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.JKSmbINum0 2024-03-16T20:18:40.815 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.JKSmbINum0 is 343KB 2024-03-16T20:18:40.874 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.JKSmbINum0 2024-03-16T20:18:40.880 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/data/mon.c.tgz 2024-03-16T20:18:40.880 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-16T20:18:40.925 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.hqokAgf0Jj 2024-03-16T20:18:40.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hqokAgf0Jj 2024-03-16T20:18:41.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.hqokAgf0Jj 2024-03-16T20:18:41.125 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.hqokAgf0Jj is 343KB 2024-03-16T20:18:41.182 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.hqokAgf0Jj 2024-03-16T20:18:41.187 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-16T20:18:41.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-16T20:18:41.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-16T20:18:41.298 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-16T20:18:41.306 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-16T20:18:41.313 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-16T20:18:41.328 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-16T20:18:41.468 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-16T20:18:41.478 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-16T20:18:41.510 INFO:tasks.ceph:Archiving crash dumps... 2024-03-16T20:18:41.512 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi032/crash 2024-03-16T20:18:41.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-16T20:18:41.526 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi059/crash 2024-03-16T20:18:41.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-16T20:18:41.540 INFO:tasks.ceph:Shutting down logrotate 2024-03-16T20:18:41.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:41.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:41.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:41.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-16T20:18:41.684 INFO:tasks.ceph:Compressing logs... 2024-03-16T20:18:41.685 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T20:18:41.706 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T20:18:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36433.log 2024-03-16T20:18:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42859.log 2024-03-16T20:18:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40554.log 2024-03-16T20:18:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36433.log: /var/log/ceph/ceph-client.admin.42859.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22781.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36433.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42859.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.22781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40554.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22781.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9%/var/log/ceph/ceph-client.admin.39031.log: -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38190.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37324.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38190.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-03-16T20:18:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35015.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38190.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5%/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37272.log 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%/var/log/ceph/ceph-client.admin.40764.log: -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34681.log 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37272.log: gzip -5 --verbose 88.5% -- /var/log/ceph/ceph-mgr.y.log -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34681.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37272.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mgr.y.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34681.log.gz 2024-03-16T20:18:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36983.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31922.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0%/var/log/ceph/ceph-client.admin.36983.log: -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33444.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31922.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36983.log.gz 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33444.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31922.log.gz 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39085.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41419.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33444.log.gz 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42623.log 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2024-03-16T20:18:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42415.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39085.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31848.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42415.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42623.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39268.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31848.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42415.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32742.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31848.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38664.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38744.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32742.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39268.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38664.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38664.log.gz 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37586.log 2024-03-16T20:18:41.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38744.log.gz 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37586.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41549.log: -- replaced with /var/log/ceph/ceph-client.admin.32742.log.gz 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37586.log.gz/var/log/ceph/ceph-client.admin.42963.log: 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40397.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41471.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40397.log: /var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39977.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40397.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32959.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35382.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40658.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32959.log: /var/log/ceph/ceph-client.admin.35382.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39977.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32959.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35382.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36223.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42545.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4%/var/log/ceph/ceph-client.admin.36223.log: -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39584.log 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36223.log.gz 2024-03-16T20:18:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42545.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42545.log.gz 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36879.log 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34008.log 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39584.log.gz/var/log/ceph/ceph-client.admin.36879.log: 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33278.log 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36879.log.gz 2024-03-16T20:18:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34008.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34008.log.gz 2024-03-16T20:18:41.755 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39452.log 2024-03-16T20:18:41.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38508.log 2024-03-16T20:18:41.755 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2%/var/log/ceph/ceph-client.admin.39452.log: -- replaced with /var/log/ceph/ceph-client.admin.33278.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33413.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39452.log.gz 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38508.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38508.log.gz 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38612.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33413.log.gz/var/log/ceph/ceph-client.admin.38612.log: 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38612.log.gz 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37875.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6%/var/log/ceph/ceph-client.admin.36774.log: -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-16T20:18:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43093.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37168.log: /var/log/ceph/ceph-client.admin.34885.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39059.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0%/var/log/ceph/ceph-client.admin.39059.log: -- replaced with /var/log/ceph/ceph-client.admin.43093.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34859.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39059.log.gz 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41627.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33492.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4%/var/log/ceph/ceph-client.admin.36037.log: -- replaced with /var/log/ceph/ceph-client.admin.34859.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.757 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33492.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33492.log.gz 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41786.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz/var/log/ceph/ceph-client.admin.41786.log: 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41786.log.gz 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41653.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38138.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42282.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5%/var/log/ceph/ceph-client.admin.38138.log: -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.758 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38138.log.gz 2024-03-16T20:18:41.759 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42282.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42282.log.gz 2024-03-16T20:18:41.765 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40083.log 2024-03-16T20:18:41.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-03-16T20:18:41.765 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%/var/log/ceph/ceph-client.admin.40083.log: -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40083.log.gz 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37194.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32083.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%/var/log/ceph/ceph-client.admin.32083.log: -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42362.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32083.log.gz 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41078.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41942.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%/var/log/ceph/ceph-client.admin.33200.log: -- replaced with /var/log/ceph/ceph-client.admin.42362.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43015.log 2024-03-16T20:18:41.766 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41942.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41942.log.gz 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6%/var/log/ceph/ceph-client.admin.38086.log: -- replaced with /var/log/ceph/ceph-client.admin.43015.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34551.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41968.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38534.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39688.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5%/var/log/ceph/ceph-client.admin.38534.log: -- replaced with /var/log/ceph/ceph-client.admin.34551.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35436.log 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38534.log.gz 2024-03-16T20:18:41.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39688.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39688.log.gz 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40265.log 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5%/var/log/ceph/ceph-client.admin.40265.log: -- replaced with /var/log/ceph/ceph-client.admin.35436.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33715.log 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40265.log.gz 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37142.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-03-16T20:18:41.768 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38403.log 2024-03-16T20:18:41.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27736.log 2024-03-16T20:18:41.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27632.log 2024-03-16T20:18:41.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27736.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27632.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27736.log.gz 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27632.log.gz 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27711.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27784.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27711.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27784.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27711.log.gz 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.x.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27784.log.gz 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27760.log 2024-03-16T20:18:41.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27687.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27760.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27808.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27687.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.27687.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27808.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27808.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27656.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.c.log: 93.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.27656.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27656.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.audit.log: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-16T20:18:41.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/valgrind/mon.c.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gzgzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr: 91.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/valgrind/mds.f.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr: 54.3% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/valgrind/mds.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/valgrind/mds.d.log: 54.3% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/valgrind/mon.b.log: 91.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-16T20:18:41.788 INFO:teuthology.orchestra.run.smithi059.stderr: 93.1% 95.1% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-16T20:18:41.789 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-16T20:18:41.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42493.log 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%/var/log/ceph/ceph-client.admin.38403.log: -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38403.log.gz 2024-03-16T20:18:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35488.log: /var/log/ceph/ceph-client.admin.42493.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42493.log.gz 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33926.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35121.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7%/var/log/ceph/ceph-client.admin.33926.log: -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33926.log.gz 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35121.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35121.log.gz 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%/var/log/ceph/ceph-client.admin.40842.log: -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31893.log 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.794 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39346.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42230.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36931.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31893.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42230.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39504.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36931.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36931.log.gz 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35226.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33491.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33089.log 2024-03-16T20:18:41.795 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33491.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33491.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35304.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43067.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35304.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35226.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33089.log: /var/log/ceph/ceph-client.admin.43067.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33089.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43067.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35304.log.gz 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36011.log 2024-03-16T20:18:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38638.log: /var/log/ceph/ceph-client.admin.37116.log: 87.8% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38560.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39899.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36011.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33305.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38560.log.gz/var/log/ceph/ceph-client.admin.39899.log: 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gzgzip 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33305.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33305.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36668.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40291.log: /var/log/ceph/ceph-client.admin.34395.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gz 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43202.log 2024-03-16T20:18:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41157.log 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36668.log: /var/log/ceph/ceph-client.admin.43202.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36668.log.gz 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43202.log.gz 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32491.log 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41838.log 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41157.log: /var/log/ceph/ceph-client.admin.32491.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41157.log.gz 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32491.log.gz 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32109.log 2024-03-16T20:18:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36905.log 2024-03-16T20:18:41.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 88.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-16T20:18:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32109.log: /var/log/ceph/ceph-client.admin.41838.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41838.log.gz 2024-03-16T20:18:41.803 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32109.log.gz 2024-03-16T20:18:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-03-16T20:18:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38586.log 2024-03-16T20:18:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36905.log: /var/log/ceph/ceph-client.admin.37716.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36905.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34161.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38586.log: /var/log/ceph/ceph-mon.a.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38586.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37953.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34161.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34161.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37953.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37953.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35907.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35174.log 2024-03-16T20:18:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42178.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2024-03-16T20:18:41.805 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32619.log 2024-03-16T20:18:41.805 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35174.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35174.log.gz 2024-03-16T20:18:41.805 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38482.log 2024-03-16T20:18:41.805 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32619.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32619.log.gz 2024-03-16T20:18:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35330.log 2024-03-16T20:18:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38482.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38482.log.gz 2024-03-16T20:18:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-03-16T20:18:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35330.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35330.log.gz 2024-03-16T20:18:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35644.log 2024-03-16T20:18:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37690.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-03-16T20:18:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33480.log 2024-03-16T20:18:41.807 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35644.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35644.log.gz 2024-03-16T20:18:41.808 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35592.log 2024-03-16T20:18:41.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33480.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33480.log.gz 2024-03-16T20:18:41.809 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2024-03-16T20:18:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35592.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35592.log.gz 2024-03-16T20:18:41.810 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36590.log 2024-03-16T20:18:41.811 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38849.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2024-03-16T20:18:41.811 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42048.log 2024-03-16T20:18:41.812 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36590.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36590.log.gz 2024-03-16T20:18:41.812 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-03-16T20:18:41.813 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42048.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42048.log.gz 2024-03-16T20:18:41.814 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40450.log 2024-03-16T20:18:41.814 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40920.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-03-16T20:18:41.815 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34107.log 2024-03-16T20:18:41.815 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40450.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40450.log.gz 2024-03-16T20:18:41.816 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35855.log 2024-03-16T20:18:41.817 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34107.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34107.log.gz 2024-03-16T20:18:41.817 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38875.log 2024-03-16T20:18:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35855.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35855.log.gz 2024-03-16T20:18:41.818 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40371.log 2024-03-16T20:18:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38875.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38875.log.gz 2024-03-16T20:18:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-03-16T20:18:41.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40371.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40371.log.gz 2024-03-16T20:18:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36091.log 2024-03-16T20:18:41.821 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36275.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-03-16T20:18:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33120.log 2024-03-16T20:18:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36091.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36091.log.gz 2024-03-16T20:18:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42334.log 2024-03-16T20:18:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33120.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33120.log.gz 2024-03-16T20:18:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40161.log 2024-03-16T20:18:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42334.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42334.log.gz 2024-03-16T20:18:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37927.log 2024-03-16T20:18:41.826 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40161.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40161.log.gz 2024-03-16T20:18:41.827 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41601.log 2024-03-16T20:18:41.828 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37927.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37927.log.gz 2024-03-16T20:18:41.828 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-03-16T20:18:41.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41601.log.gz 2024-03-16T20:18:41.829 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-03-16T20:18:41.830 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-03-16T20:18:41.831 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42571.log 2024-03-16T20:18:41.832 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34265.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz 2024-03-16T20:18:41.832 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38059.log 2024-03-16T20:18:41.832 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33790.log 2024-03-16T20:18:41.833 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42571.log: 88.5%/var/log/ceph/ceph-client.admin.38059.log: -- replaced with /var/log/ceph/ceph-client.admin.42571.log.gz 2024-03-16T20:18:41.834 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37560.log 2024-03-16T20:18:41.835 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38059.log.gz 2024-03-16T20:18:41.835 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33790.log.gz 2024-03-16T20:18:41.835 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37797.log 2024-03-16T20:18:41.835 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42701.log 2024-03-16T20:18:41.836 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37560.log: /var/log/ceph/ceph-client.admin.37797.log: 88.9% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37797.log.gz 2024-03-16T20:18:41.836 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37560.log.gz 2024-03-16T20:18:41.836 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-03-16T20:18:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36143.log 2024-03-16T20:18:41.838 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42701.log: /var/log/ceph/ceph-client.admin.40972.log: 88.0% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42701.log.gz 2024-03-16T20:18:41.838 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-03-16T20:18:41.838 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33063.log 2024-03-16T20:18:41.838 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34057.log 2024-03-16T20:18:41.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-03-16T20:18:41.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34057.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36143.log.gz 2024-03-16T20:18:41.839 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-03-16T20:18:41.839 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34057.log.gz 2024-03-16T20:18:41.840 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39372.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-03-16T20:18:41.840 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-03-16T20:18:41.841 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37404.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-03-16T20:18:41.841 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32543.log 2024-03-16T20:18:41.842 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39426.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-03-16T20:18:41.842 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37770.log 2024-03-16T20:18:41.843 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32543.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32543.log.gz 2024-03-16T20:18:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34343.log 2024-03-16T20:18:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37770.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37770.log.gz 2024-03-16T20:18:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36407.log 2024-03-16T20:18:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34343.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2024-03-16T20:18:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-03-16T20:18:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36407.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36407.log.gz 2024-03-16T20:18:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-03-16T20:18:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39873.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-03-16T20:18:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34291.log 2024-03-16T20:18:41.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-03-16T20:18:41.850 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34989.log 2024-03-16T20:18:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34291.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34291.log.gz 2024-03-16T20:18:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42441.log 2024-03-16T20:18:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34989.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34989.log.gz 2024-03-16T20:18:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-03-16T20:18:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42441.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42441.log.gz 2024-03-16T20:18:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32387.log 2024-03-16T20:18:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38927.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-03-16T20:18:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42805.log 2024-03-16T20:18:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32387.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32387.log.gz 2024-03-16T20:18:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-03-16T20:18:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42805.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42805.log.gz 2024-03-16T20:18:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-03-16T20:18:41.859 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33063.log.gz 2024-03-16T20:18:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40998.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-03-16T20:18:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2024-03-16T20:18:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40109.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-03-16T20:18:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33172.log 2024-03-16T20:18:41.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39190.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-03-16T20:18:41.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-03-16T20:18:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33172.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33172.log.gz 2024-03-16T20:18:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36486.log 2024-03-16T20:18:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34807.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2024-03-16T20:18:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32932.log 2024-03-16T20:18:41.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36486.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36486.log.gz 2024-03-16T20:18:41.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-16T20:18:41.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32932.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32932.log.gz 2024-03-16T20:18:41.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-03-16T20:18:41.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-16T20:18:41.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38979.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-03-16T20:18:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-03-16T20:18:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41105.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34937.log 2024-03-16T20:18:41.895 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-16T20:18:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-03-16T20:18:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-03-16T20:18:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34937.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34937.log.gz 2024-03-16T20:18:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35462.log 2024-03-16T20:18:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37979.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-03-16T20:18:41.925 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32439.log 2024-03-16T20:18:41.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35462.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35462.log.gz 2024-03-16T20:18:41.930 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33665.log 2024-03-16T20:18:41.931 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32439.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32439.log.gz 2024-03-16T20:18:41.932 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40606.log 2024-03-16T20:18:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33671.log 2024-03-16T20:18:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40606.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40606.log.gz 2024-03-16T20:18:41.945 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33665.log.gz 2024-03-16T20:18:41.947 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-16T20:18:41.948 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-03-16T20:18:41.949 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33671.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33671.log.gz 2024-03-16T20:18:41.949 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2024-03-16T20:18:41.950 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35959.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-03-16T20:18:41.950 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2024-03-16T20:18:41.951 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41235.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2024-03-16T20:18:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2024-03-16T20:18:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42074.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2024-03-16T20:18:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42833.log 2024-03-16T20:18:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34709.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz 2024-03-16T20:18:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40424.log 2024-03-16T20:18:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42833.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42833.log.gz 2024-03-16T20:18:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39005.log 2024-03-16T20:18:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40424.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40424.log.gz 2024-03-16T20:18:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35095.log 2024-03-16T20:18:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39005.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39005.log.gz 2024-03-16T20:18:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36195.log 2024-03-16T20:18:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35095.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35095.log.gz 2024-03-16T20:18:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36642.log 2024-03-16T20:18:41.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36195.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36195.log.gz 2024-03-16T20:18:41.961 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37350.log 2024-03-16T20:18:41.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36642.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36642.log.gz 2024-03-16T20:18:41.962 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-03-16T20:18:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37350.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37350.log.gz 2024-03-16T20:18:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38323.log 2024-03-16T20:18:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42779.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-03-16T20:18:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32569.log 2024-03-16T20:18:41.965 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38323.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38323.log.gz 2024-03-16T20:18:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32880.log 2024-03-16T20:18:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-03-16T20:18:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32569.log: /var/log/ceph/ceph-client.admin.32880.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32880.log.gz 2024-03-16T20:18:41.967 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.32569.log.gz 2024-03-16T20:18:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32985.log 2024-03-16T20:18:41.968 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35933.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-03-16T20:18:41.968 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37063.log 2024-03-16T20:18:41.969 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32985.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32985.log.gz 2024-03-16T20:18:41.970 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35540.log 2024-03-16T20:18:41.971 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37063.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37063.log.gz 2024-03-16T20:18:41.971 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33763.log 2024-03-16T20:18:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-03-16T20:18:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35540.log: /var/log/ceph/ceph-client.admin.33763.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35540.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33763.log.gz 2024-03-16T20:18:41.973 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-16T20:18:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39294.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-03-16T20:18:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40684.log 2024-03-16T20:18:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38242.log 2024-03-16T20:18:41.990 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40684.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2024-03-16T20:18:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40239.log 2024-03-16T20:18:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38242.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38242.log.gz 2024-03-16T20:18:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-16T20:18:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40239.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40239.log.gz 2024-03-16T20:18:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37901.log 2024-03-16T20:18:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39558.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-16T20:18:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-03-16T20:18:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37901.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37901.log.gz 2024-03-16T20:18:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-03-16T20:18:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41733.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-03-16T20:18:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37744.log 2024-03-16T20:18:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38005.log 2024-03-16T20:18:42.061 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-03-16T20:18:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37744.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37744.log.gz 2024-03-16T20:18:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35566.log 2024-03-16T20:18:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38005.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38005.log.gz 2024-03-16T20:18:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35041.log 2024-03-16T20:18:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35566.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35566.log.gz 2024-03-16T20:18:42.097 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-03-16T20:18:42.098 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35041.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35041.log.gz 2024-03-16T20:18:42.109 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2024-03-16T20:18:42.110 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37612.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-03-16T20:18:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-03-16T20:18:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40738.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2024-03-16T20:18:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33011.log 2024-03-16T20:18:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38216.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-03-16T20:18:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-03-16T20:18:42.134 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33011.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33011.log.gz 2024-03-16T20:18:42.145 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35670.log 2024-03-16T20:18:42.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33490.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-03-16T20:18:42.157 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40476.log 2024-03-16T20:18:42.158 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35670.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35670.log.gz 2024-03-16T20:18:42.169 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38112.log 2024-03-16T20:18:42.170 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40476.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40476.log.gz 2024-03-16T20:18:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32465.log 2024-03-16T20:18:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38112.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38112.log.gz 2024-03-16T20:18:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31786.log 2024-03-16T20:18:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32465.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32465.log.gz 2024-03-16T20:18:42.200 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-16T20:18:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33930.log 2024-03-16T20:18:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31786.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31786.log.gz 2024-03-16T20:18:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-03-16T20:18:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33930.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33930.log.gz 2024-03-16T20:18:42.232 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-03-16T20:18:42.233 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37430.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-03-16T20:18:42.239 INFO:teuthology.orchestra.run.smithi059.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-16T20:18:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-03-16T20:18:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38375.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-03-16T20:18:42.265 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32242.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-03-16T20:18:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39400.log 2024-03-16T20:18:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2024-03-16T20:18:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39400.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39400.log.gz 2024-03-16T20:18:42.293 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38164.log 2024-03-16T20:18:42.294 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38716.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2024-03-16T20:18:42.305 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34755.log 2024-03-16T20:18:42.306 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38164.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38164.log.gz 2024-03-16T20:18:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39951.log 2024-03-16T20:18:42.310 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34755.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34755.log.gz 2024-03-16T20:18:42.321 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-03-16T20:18:42.322 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39951.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2024-03-16T20:18:42.337 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-03-16T20:18:42.338 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41052.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-03-16T20:18:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-03-16T20:18:42.350 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35985.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-03-16T20:18:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-16T20:18:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34963.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-03-16T20:18:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35356.log 2024-03-16T20:18:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40868.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-03-16T20:18:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41367.log 2024-03-16T20:18:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35356.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35356.log.gz 2024-03-16T20:18:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35514.log 2024-03-16T20:18:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41367.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41367.log.gz 2024-03-16T20:18:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37823.log 2024-03-16T20:18:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35514.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35514.log.gz 2024-03-16T20:18:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34629.log 2024-03-16T20:18:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37823.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37823.log.gz 2024-03-16T20:18:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-16T20:18:42.450 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34629.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34629.log.gz 2024-03-16T20:18:42.465 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31995.log 2024-03-16T20:18:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: 88.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-16T20:18:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36117.log 2024-03-16T20:18:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-03-16T20:18:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36117.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36117.log.gz 2024-03-16T20:18:42.493 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31995.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31995.log.gz 2024-03-16T20:18:42.497 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37009.log 2024-03-16T20:18:42.498 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42388.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-03-16T20:18:42.505 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34499.log 2024-03-16T20:18:42.506 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37009.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37009.log.gz 2024-03-16T20:18:42.533 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-03-16T20:18:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34499.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34499.log.gz 2024-03-16T20:18:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34131.log 2024-03-16T20:18:42.535 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39242.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-03-16T20:18:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37089.log 2024-03-16T20:18:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34131.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34131.log.gz 2024-03-16T20:18:42.593 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32517.log 2024-03-16T20:18:42.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37089.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37089.log.gz 2024-03-16T20:18:42.595 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37456.log 2024-03-16T20:18:42.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32517.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32517.log.gz 2024-03-16T20:18:42.607 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36327.log 2024-03-16T20:18:42.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37456.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37456.log.gz 2024-03-16T20:18:42.608 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42256.log 2024-03-16T20:18:42.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36327.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36327.log.gz 2024-03-16T20:18:42.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-03-16T20:18:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42727.log 2024-03-16T20:18:42.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38429.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-03-16T20:18:42.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42256.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42256.log.gz 2024-03-16T20:18:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38690.log 2024-03-16T20:18:42.638 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42727.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42727.log.gz 2024-03-16T20:18:42.646 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41131.log 2024-03-16T20:18:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38690.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38690.log.gz 2024-03-16T20:18:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35410.log 2024-03-16T20:18:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41131.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41131.log.gz 2024-03-16T20:18:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-03-16T20:18:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35410.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35410.log.gz 2024-03-16T20:18:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34447.log 2024-03-16T20:18:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39742.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-03-16T20:18:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34525.log 2024-03-16T20:18:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34447.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34447.log.gz 2024-03-16T20:18:42.718 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35278.log 2024-03-16T20:18:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34525.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-03-16T20:18:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33685.log 2024-03-16T20:18:42.734 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35278.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35278.log.gz 2024-03-16T20:18:42.749 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41760.log 2024-03-16T20:18:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33685.log.gz 2024-03-16T20:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34033.log 2024-03-16T20:18:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41760.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41760.log.gz 2024-03-16T20:18:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34145.log 2024-03-16T20:18:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34033.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34033.log.gz 2024-03-16T20:18:42.789 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-03-16T20:18:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34145.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34145.log.gz 2024-03-16T20:18:42.793 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33252.log 2024-03-16T20:18:42.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41575.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-03-16T20:18:42.801 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-03-16T20:18:42.802 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33252.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33252.log.gz 2024-03-16T20:18:42.813 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-16T20:18:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39925.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-03-16T20:18:42.821 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-16T20:18:42.822 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40790.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-16T20:18:42.837 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34317.log 2024-03-16T20:18:42.838 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40003.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-16T20:18:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42937.log 2024-03-16T20:18:42.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34317.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34317.log.gz 2024-03-16T20:18:42.846 INFO:teuthology.orchestra.run.smithi059.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-16T20:18:42.853 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2024-03-16T20:18:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42937.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42937.log.gz 2024-03-16T20:18:42.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34421.log 2024-03-16T20:18:42.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42467.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2024-03-16T20:18:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32854.log 2024-03-16T20:18:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34421.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34421.log.gz 2024-03-16T20:18:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35722.log 2024-03-16T20:18:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32854.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32854.log.gz 2024-03-16T20:18:42.900 INFO:teuthology.orchestra.run.smithi059.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-16T20:18:42.901 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32906.log 2024-03-16T20:18:42.902 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35722.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35722.log.gz 2024-03-16T20:18:42.909 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-03-16T20:18:42.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32906.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32906.log.gz 2024-03-16T20:18:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-16T20:18:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-03-16T20:18:42.918 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35776.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-03-16T20:18:42.925 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-03-16T20:18:42.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42989.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2024-03-16T20:18:42.933 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-03-16T20:18:42.934 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40187.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-03-16T20:18:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-03-16T20:18:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39610.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-03-16T20:18:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42519.log 2024-03-16T20:18:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35200.log: 89.4% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-03-16T20:18:42.951 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-16T20:18:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41864.log 2024-03-16T20:18:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42519.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42519.log.gz 2024-03-16T20:18:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2024-03-16T20:18:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41864.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41864.log.gz 2024-03-16T20:18:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33383.log 2024-03-16T20:18:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39164.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2024-03-16T20:18:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-16T20:18:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33383.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33383.log.gz 2024-03-16T20:18:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33489.log 2024-03-16T20:18:42.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40946.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-16T20:18:42.957 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33684.log 2024-03-16T20:18:42.958 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33489.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33489.log.gz 2024-03-16T20:18:42.958 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36853.log 2024-03-16T20:18:42.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33684.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33684.log.gz 2024-03-16T20:18:42.960 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33496.log 2024-03-16T20:18:42.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36853.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36853.log.gz 2024-03-16T20:18:42.960 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32355.log 2024-03-16T20:18:42.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33496.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33496.log.gz 2024-03-16T20:18:42.962 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34473.log 2024-03-16T20:18:42.962 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32355.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.32355.log.gz 2024-03-16T20:18:42.963 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33331.log 2024-03-16T20:18:42.964 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34473.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34473.log.gz 2024-03-16T20:18:42.964 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36564.log 2024-03-16T20:18:42.964 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32190.log 2024-03-16T20:18:42.965 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36564.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36564.log.gz 2024-03-16T20:18:42.966 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33331.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33331.log.gz 2024-03-16T20:18:42.967 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36616.log 2024-03-16T20:18:42.968 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32190.log.gz 2024-03-16T20:18:42.968 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40317.log 2024-03-16T20:18:42.969 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36616.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36616.log.gz 2024-03-16T20:18:42.969 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-03-16T20:18:42.970 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40317.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40317.log.gz 2024-03-16T20:18:42.970 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35750.log 2024-03-16T20:18:42.971 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41024.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-03-16T20:18:42.972 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-03-16T20:18:42.972 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35750.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35750.log.gz 2024-03-16T20:18:42.973 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39112.log 2024-03-16T20:18:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41523.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-03-16T20:18:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35881.log 2024-03-16T20:18:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39112.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27588.log 2024-03-16T20:18:42.976 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35881.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35881.log.gz 2024-03-16T20:18:42.976 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2024-03-16T20:18:42.977 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34369.log 2024-03-16T20:18:42.977 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.27588.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.27588.log.gz 2024-03-16T20:18:42.978 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-16T20:18:42.978 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34369.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34369.log.gz 2024-03-16T20:18:42.979 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-16T20:18:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-16T20:18:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32320.log 2024-03-16T20:18:43.008 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32826.log 2024-03-16T20:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32320.log.gz 2024-03-16T20:18:43.021 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-16T20:18:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32826.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32826.log.gz 2024-03-16T20:18:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping 2024-03-16T20:18:43.032 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-16T20:18:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39662.log 2024-03-16T20:18:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40213.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-16T20:18:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-16T20:18:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39662.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39662.log.gz 2024-03-16T20:18:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39478.log 2024-03-16T20:18:43.045 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.a.log: 92.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-16T20:18:43.046 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38349.log 2024-03-16T20:18:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39478.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39478.log.gz 2024-03-16T20:18:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42649.log 2024-03-16T20:18:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38349.log.gz 2024-03-16T20:18:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-03-16T20:18:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42649.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2024-03-16T20:18:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33226.log 2024-03-16T20:18:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38901.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-03-16T20:18:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-03-16T20:18:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33226.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33226.log.gz 2024-03-16T20:18:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32800.log 2024-03-16T20:18:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41393.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-03-16T20:18:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33037.log 2024-03-16T20:18:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32800.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32800.log.gz 2024-03-16T20:18:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41916.log 2024-03-16T20:18:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33037.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33037.log.gz 2024-03-16T20:18:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-03-16T20:18:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41916.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41916.log.gz 2024-03-16T20:18:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36957.log 2024-03-16T20:18:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-03-16T20:18:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33357.log 2024-03-16T20:18:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36957.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36957.log.gz 2024-03-16T20:18:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32294.log 2024-03-16T20:18:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33357.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33357.log.gz 2024-03-16T20:18:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-16T20:18:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32294.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32294.log.gz 2024-03-16T20:18:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-16T20:18:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mds.a.log: 54.3% -- replaced with /var/log/ceph/valgrind/mds.a.log.gz 2024-03-16T20:18:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-16T20:18:43.060 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mon.a.log: 91.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-16T20:18:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-03-16T20:18:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mds.c.log: 54.3% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-16T20:18:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38268.log 2024-03-16T20:18:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mds.e.log: 54.3% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-16T20:18:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-03-16T20:18:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38268.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38268.log.gz 2024-03-16T20:18:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43041.log 2024-03-16T20:18:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39795.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-03-16T20:18:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36169.log 2024-03-16T20:18:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43041.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43041.log.gz 2024-03-16T20:18:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39847.log 2024-03-16T20:18:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36169.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36169.log.gz 2024-03-16T20:18:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37376.log 2024-03-16T20:18:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39847.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39847.log.gz 2024-03-16T20:18:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35252.log 2024-03-16T20:18:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37376.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37376.log.gz 2024-03-16T20:18:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36512.log 2024-03-16T20:18:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35252.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35252.log.gz 2024-03-16T20:18:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35829.log 2024-03-16T20:18:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36512.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36512.log.gz 2024-03-16T20:18:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-03-16T20:18:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35829.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35829.log.gz 2024-03-16T20:18:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33984.log 2024-03-16T20:18:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38823.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-03-16T20:18:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2024-03-16T20:18:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33984.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33984.log.gz 2024-03-16T20:18:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43173.log 2024-03-16T20:18:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40894.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2024-03-16T20:18:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34205.log 2024-03-16T20:18:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43173.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43173.log.gz 2024-03-16T20:18:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37482.log 2024-03-16T20:18:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34205.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34205.log.gz 2024-03-16T20:18:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-03-16T20:18:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37482.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37482.log.gz 2024-03-16T20:18:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37035.log 2024-03-16T20:18:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33896.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-03-16T20:18:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42753.log 2024-03-16T20:18:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37035.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37035.log.gz 2024-03-16T20:18:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41209.log 2024-03-16T20:18:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42753.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42753.log.gz 2024-03-16T20:18:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2024-03-16T20:18:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41209.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41209.log.gz 2024-03-16T20:18:43.082 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42885.log 2024-03-16T20:18:43.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41812.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2024-03-16T20:18:43.083 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2024-03-16T20:18:43.084 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42885.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42885.log.gz 2024-03-16T20:18:43.084 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36353.log 2024-03-16T20:18:43.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36800.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2024-03-16T20:18:43.086 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-03-16T20:18:43.086 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36353.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36353.log.gz 2024-03-16T20:18:43.087 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33498.log 2024-03-16T20:18:43.087 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40135.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-03-16T20:18:43.087 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36460.log 2024-03-16T20:18:43.088 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33498.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33498.log.gz 2024-03-16T20:18:43.089 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32138.log 2024-03-16T20:18:43.089 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36460.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36460.log.gz 2024-03-16T20:18:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2024-03-16T20:18:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32138.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32138.log.gz 2024-03-16T20:18:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36748.log 2024-03-16T20:18:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42204.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz 2024-03-16T20:18:43.092 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32052.log 2024-03-16T20:18:43.092 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36748.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36748.log.gz 2024-03-16T20:18:43.093 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-03-16T20:18:43.094 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32052.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32052.log.gz 2024-03-16T20:18:43.094 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38797.log 2024-03-16T20:18:43.095 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41445.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-03-16T20:18:43.095 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2024-03-16T20:18:43.096 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38797.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38797.log.gz 2024-03-16T20:18:43.096 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31948.log 2024-03-16T20:18:43.097 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40528.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gz 2024-03-16T20:18:43.097 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-03-16T20:18:43.098 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.31948.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31948.log.gz 2024-03-16T20:18:43.098 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43226.log 2024-03-16T20:18:43.099 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37664.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-03-16T20:18:43.099 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-03-16T20:18:43.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43226.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43226.log.gz 2024-03-16T20:18:43.100 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-03-16T20:18:43.101 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42308.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-03-16T20:18:43.101 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-03-16T20:18:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40029.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-03-16T20:18:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33739.log 2024-03-16T20:18:43.103 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37534.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-03-16T20:18:43.103 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-16T20:18:43.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33739.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33739.log.gz 2024-03-16T20:18:43.104 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27901.log 2024-03-16T20:18:43.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41261.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-16T20:18:43.106 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38456.log 2024-03-16T20:18:43.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.27901.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27901.log.gz 2024-03-16T20:18:43.107 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41994.log 2024-03-16T20:18:43.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38456.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38456.log.gz 2024-03-16T20:18:43.108 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34833.log 2024-03-16T20:18:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41994.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41994.log.gz 2024-03-16T20:18:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34781.log 2024-03-16T20:18:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34833.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34833.log.gz 2024-03-16T20:18:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39530.log 2024-03-16T20:18:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34781.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34781.log.gz 2024-03-16T20:18:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-03-16T20:18:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39530.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39530.log.gz 2024-03-16T20:18:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2024-03-16T20:18:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39138.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-03-16T20:18:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39716.log 2024-03-16T20:18:43.115 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42126.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2024-03-16T20:18:43.116 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-16T20:18:43.116 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39716.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39716.log.gz 2024-03-16T20:18:43.117 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37220.log 2024-03-16T20:18:43.118 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-16T20:18:43.118 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2024-03-16T20:18:43.119 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37220.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37220.log.gz 2024-03-16T20:18:43.119 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42020.log 2024-03-16T20:18:43.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38031.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2024-03-16T20:18:43.121 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-03-16T20:18:43.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42020.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42020.log.gz 2024-03-16T20:18:43.122 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-16T20:18:43.123 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36720.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-03-16T20:18:43.123 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32216.log 2024-03-16T20:18:43.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40712.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-16T20:18:43.124 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-03-16T20:18:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32216.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32216.log.gz 2024-03-16T20:18:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36379.log 2024-03-16T20:18:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35067.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz 2024-03-16T20:18:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32164.log 2024-03-16T20:18:43.127 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-16T20:18:43.128 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36379.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36379.log.gz 2024-03-16T20:18:43.128 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-03-16T20:18:43.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32164.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32164.log.gz 2024-03-16T20:18:43.130 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-03-16T20:18:43.130 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-16T20:18:43.130 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m1.392s 2024-03-16T20:18:43.131 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m5.510s 2024-03-16T20:18:43.131 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.199s 2024-03-16T20:18:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38771.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-03-16T20:18:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36065.log 2024-03-16T20:18:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40502.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-03-16T20:18:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41707.log 2024-03-16T20:18:43.133 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36065.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36065.log.gz 2024-03-16T20:18:43.133 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37508.log 2024-03-16T20:18:43.134 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41707.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41707.log.gz 2024-03-16T20:18:43.134 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34577.log 2024-03-16T20:18:43.135 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37508.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37508.log.gz 2024-03-16T20:18:43.136 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-03-16T20:18:43.136 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34577.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34577.log.gz 2024-03-16T20:18:43.137 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32413.log 2024-03-16T20:18:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40580.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-03-16T20:18:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36301.log 2024-03-16T20:18:43.139 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32413.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32413.log.gz 2024-03-16T20:18:43.139 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-16T20:18:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36301.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36301.log.gz 2024-03-16T20:18:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42597.log 2024-03-16T20:18:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39636.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-16T20:18:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32268.log 2024-03-16T20:18:43.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42597.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42597.log.gz 2024-03-16T20:18:43.143 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33146.log 2024-03-16T20:18:43.144 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.32268.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32268.log.gz 2024-03-16T20:18:43.144 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37849.log 2024-03-16T20:18:43.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33146.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33146.log.gz 2024-03-16T20:18:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2024-03-16T20:18:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37849.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37849.log.gz 2024-03-16T20:18:43.147 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33920.log 2024-03-16T20:18:43.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42152.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2024-03-16T20:18:43.148 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34209.log 2024-03-16T20:18:43.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.33920.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33920.log.gz 2024-03-16T20:18:43.149 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-16T20:18:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34209.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34209.log.gz 2024-03-16T20:18:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35803.log 2024-03-16T20:18:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.e.log: 94.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2024-03-16T20:18:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35803.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35803.log.gz 2024-03-16T20:18:43.165 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-16T20:18:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36827.log 2024-03-16T20:18:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41287.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2024-03-16T20:18:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 2024-03-16T20:18:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36827.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36827.log.gz 2024-03-16T20:18:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-03-16T20:18:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41497.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-03-16T20:18:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-03-16T20:18:43.170 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42100.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-03-16T20:18:43.170 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35696.log 2024-03-16T20:18:43.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34655.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-03-16T20:18:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-03-16T20:18:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35696.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35696.log.gz 2024-03-16T20:18:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-03-16T20:18:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35618.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-03-16T20:18:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37246.log 2024-03-16T20:18:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43145.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-03-16T20:18:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27788.log 2024-03-16T20:18:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37246.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37246.log.gz 2024-03-16T20:18:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36694.log 2024-03-16T20:18:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.27788.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27788.log.gz 2024-03-16T20:18:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34603.log 2024-03-16T20:18:43.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36694.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36694.log.gz 2024-03-16T20:18:43.179 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36538.log 2024-03-16T20:18:43.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34603.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34603.log.gz 2024-03-16T20:18:43.180 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40816.log 2024-03-16T20:18:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36538.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36538.log.gz 2024-03-16T20:18:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37298.log 2024-03-16T20:18:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40816.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40816.log.gz 2024-03-16T20:18:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40632.log 2024-03-16T20:18:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37298.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37298.log.gz 2024-03-16T20:18:43.184 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36249.log 2024-03-16T20:18:43.185 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40632.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40632.log.gz 2024-03-16T20:18:43.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36249.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36249.log.gz 2024-03-16T20:18:43.216 INFO:teuthology.orchestra.run.smithi032.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-16T20:18:43.226 INFO:teuthology.orchestra.run.smithi032.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-16T20:18:43.376 INFO:teuthology.orchestra.run.smithi032.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-16T20:18:43.407 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-16T20:18:43.409 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-16T20:18:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m1.701s 2024-03-16T20:18:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m7.271s 2024-03-16T20:18:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.378s 2024-03-16T20:18:43.410 INFO:tasks.ceph:Archiving logs... 2024-03-16T20:18:43.410 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi032/log 2024-03-16T20:18:43.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T20:18:44.411 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi059/log 2024-03-16T20:18:44.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T20:18:44.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~41s 2024-03-16T20:18:44.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-16T20:18:44.997 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-16T20:18:45.224 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1e56dfe943844e92862de720010fe081 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-16T20:18:45.228 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-16T20:18:45.240 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T20:18:45.282 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T20:18:45.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T20:18:45.296 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T20:18:45.360 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T20:18:45.368 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T20:18:45.369 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T20:18:45.371 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T20:18:45.379 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-03-16T20:18:45.379 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-03-16T20:18:45.379 INFO:teuthology.orchestra.run.smithi032.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.065 -5.081 1.244 2024-03-16T20:18:45.379 INFO:teuthology.orchestra.run.smithi032.stdout:+hv02.front.sepi 63.231.80.2 3 u 9 64 377 0.085 -3.072 0.722 2024-03-16T20:18:45.380 INFO:teuthology.orchestra.run.smithi032.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.096 -2.396 0.701 2024-03-16T20:18:45.380 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout:*hv01.front.sepi 44.4.53.2 2 u 27 128 377 0.082 -1.853 0.285 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout:+hv02.front.sepi 63.231.80.2 3 u 76 128 377 0.068 -1.279 0.154 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout:+hv03.front.sepi 74.6.168.72 3 u 76 128 377 0.071 -0.679 0.407 2024-03-16T20:18:45.383 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-16T20:18:45.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T20:18:45.399 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T20:18:45.400 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T20:18:45.441 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T20:18:45.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T20:18:45.524 INFO:teuthology.task.internal:Duration was 2351.839854 seconds 2024-03-16T20:18:45.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T20:18:45.567 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T20:18:45.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T20:18:45.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T20:18:45.603 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T20:18:45.603 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-03-16T20:18:45.604 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T20:18:45.646 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-03-16T20:18:45.646 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T20:18:45.655 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T20:18:45.656 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T20:18:45.690 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T20:18:45.720 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T20:18:45.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T20:18:45.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T20:18:45.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T20:18:45.836 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T20:18:45.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T20:18:45.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T20:18:45.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T20:18:45.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T20:18:45.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T20:18:45.897 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-03-16T20:18:45.911 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-03-16T20:18:45.923 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T20:18:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:45.952 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T20:18:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:45.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T20:18:45.977 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T20:18:45.978 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi032 2024-03-16T20:18:45.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T20:18:46.010 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118/remote/smithi059 2024-03-16T20:18:46.011 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T20:18:46.032 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T20:18:46.033 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T20:18:46.050 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T20:18:46.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T20:18:46.088 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T20:18:46.088 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T20:18:46.118 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T20:18:46.118 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T20:18:46.120 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T20:18:46.123 INFO:teuthology.orchestra.run.smithi032.stdout: 658617 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 20:18 /home/ubuntu/cephtest 2024-03-16T20:18:46.125 INFO:teuthology.orchestra.run.smithi059.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 20:18 /home/ubuntu/cephtest 2024-03-16T20:18:46.127 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T20:18:46.159 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T20:18:46.243 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T20:18:46.270 DEBUG:teuthology.nuke:shortname: smithi032 2024-03-16T20:18:46.270 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T20:18:46.294 DEBUG:teuthology.nuke:shortname: smithi059 2024-03-16T20:18:46.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T20:18:46.298 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:31:07.741885', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZw9i/4Vc9mop3CJjhqkAmxhPoQx9KrKTXVtPi6nmyvo7tsBHjJcg/SGsfC7PUNI7tjtIVIhxU0ZRID3L2wjFk='} 2024-03-16T20:18:46.317 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:31:07.739957', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtaE+d2gd3u/ZI1NLSrq7vqbsLHzfwJBx7MfraW5/9rpHJZc1isETXaFsfab+PC6JveTnZ0pPQTlW8a5YAxKgY='} 2024-03-16T20:18:46.350 INFO:teuthology.orchestra.console.smithi032:Power off 2024-03-16T20:18:46.350 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T20:18:46.368 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-16T20:18:46.368 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T20:18:46.377 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-03-16T20:18:46.378 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:46.391 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-16T20:18:46.391 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:46.401 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-16T20:18:46.414 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-16T20:18:49.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-16T20:18:49.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-16T20:18:49.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-03-16T20:18:49.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~46s 2024-03-16T20:18:49.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-16T20:18:49.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-16T20:18:50.402 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:50.415 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:50.514 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-16T20:18:50.528 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-16T20:18:54.519 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:54.528 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:54.631 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-03-16T20:18:54.639 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-16T20:18:54.732 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-03-16T20:18:54.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2024-03-16T20:18:54.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-03-16T20:18:54.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~5s 2024-03-16T20:18:54.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~51s 2024-03-16T20:18:54.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2024-03-16T20:18:54.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~5s 2024-03-16T20:18:58.642 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T20:18:58.755 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-16T20:18:58.856 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-16T20:18:58.961 INFO:teuthology.run:Summary data: description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} duration: 2351.8398537635803 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1e56dfe943844e92862de720010fe081 status: fail success: false 2024-03-16T20:18:58.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T20:18:59.059 INFO:teuthology.run:FAIL